You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by David Ryan <da...@livemedia.com.au> on 2020/04/10 06:30:32 UTC
Re: Lucene 9.0 Java module system support
Hi Uwe,
As I mentioned in the original post, the main issue I've come across with
being able to support Java 11 modules is that Lucene doesn't cleanly
separate the use of package names across jar libraries which results in
errors like:
"The package org.apache.lucene.analysis.standard is accessible from more
than one module: lucene.analyzers.common, lucene.core"
I ran some analysis to see how extensive the problem is using the code
below (pass in the directory of unzipped Lucene 8.5.0 as argument).
Actually, the problem is really not that big and could be fixed. If you
ignore duplicates in lucene-test-framework-8.5.0.jar,
lucene-backward-codecs-8.5.0.jar and lucene-misc-8.5.0, you are left with
the following issues:
package org.apache.lucene.analysis.standard
- lucene-core-8.5.0.jar
- lucene-analyzers-common-8.5.0.jar
Would it be viable to rename package org.apache.lucene.analysis.standard in
lucene-analyzers-common to org.apache.lucene.analysis.classic or move the
classes into the core library?
package org.apache.lucene.search
package org.apache.lucene.document
- lucene-core-8.5.0.jar
- lucene-sandbox-8.5.0.jar
Would it be possible to move the search and document package in
lucene-sandbox into a sandbox sub-package? e.g.
org.apache.lucene.sandbox.search
package org.apache.lucene.collation
package org.apache.lucene.collation.tokenattributes
- lucene-analyzers-common-8.5.0.jar
- lucene-analyzers-icu-8.5.0.jar
Would it be possible to move collation into a collation.icu sub-package?
e.g. org.apache.lucene.collation.icu
As I'm not deeply familiar with the Lucene code base, I'm don't know what
the flow-on effects of these changes would be. However, I'd be happy to
raise the JIRA issue(s) and prepare the changes/pull request.
I believe with these changes many of the important Lucene libraries could
be cleanly brought into an application using the module system. Another
nice to have would be to add an Automatic-Module-Name entry to the
MANIFEST.MF of each jar to stop automated module names.
As you mentioned the fall back is to create a lucene-all-8.5.0.jar and
combine all classes into a single large file. Given this is not currently
published on Maven, if there's no interest in making the above changes,
would it be possible to change the build system to publish a lucene-all
package?
Regards,
David.
-------- Full output of duplicated packages --------
package org.apache.lucene.analysis.standard
- lucene-core-8.5.0.jar
- lucene-analyzers-common-8.5.0.jar
- lucene-test-framework-8.5.0.jar
package org.apache.lucene.codecs.blockterms
- lucene-codecs-8.5.0.jar
- lucene-test-framework-8.5.0.jar
package org.apache.lucene.codecs.lucene80
- lucene-core-8.5.0.jar
- lucene-backward-codecs-8.5.0.jar
package org.apache.lucene.search.similarities
- lucene-core-8.5.0.jar
- lucene-test-framework-8.5.0.jar
package org.apache.lucene.codecs.lucene70
- lucene-core-8.5.0.jar
- lucene-backward-codecs-8.5.0.jar
package org.apache.lucene.store
- lucene-misc-8.5.0.jar
- lucene-core-8.5.0.jar
- lucene-test-framework-8.5.0.jar
package org.apache.lucene.codecs.bloom
- lucene-codecs-8.5.0.jar
- lucene-test-framework-8.5.0.jar
package org.apache.lucene.codecs.lucene50
- lucene-core-8.5.0.jar
- lucene-backward-codecs-8.5.0.jar
package org.apache.lucene.search.spans
- lucene-core-8.5.0.jar
- lucene-test-framework-8.5.0.jar
package org.apache.lucene.index
- lucene-misc-8.5.0.jar
- lucene-core-8.5.0.jar
- lucene-test-framework-8.5.0.jar
package org.apache.lucene.util.fst
- lucene-misc-8.5.0.jar
- lucene-core-8.5.0.jar
- lucene-test-framework-8.5.0.jar
package org.apache.lucene.analysis
- lucene-core-8.5.0.jar
- lucene-test-framework-8.5.0.jar
package org.apache.lucene.collation
- lucene-analyzers-common-8.5.0.jar
- lucene-analyzers-icu-8.5.0.jar
package org.apache.lucene.codecs.uniformsplit.sharedterms
- lucene-codecs-8.5.0.jar
- lucene-test-framework-8.5.0.jar
package org.apache.lucene.geo
- lucene-core-8.5.0.jar
- lucene-test-framework-8.5.0.jar
package org.apache.lucene.search
- lucene-misc-8.5.0.jar
- lucene-core-8.5.0.jar
- lucene-test-framework-8.5.0.jar
- lucene-sandbox-8.5.0.jar
package org.apache.lucene.collation.tokenattributes
- lucene-analyzers-common-8.5.0.jar
- lucene-analyzers-icu-8.5.0.jar
package org.apache.lucene.document
- lucene-misc-8.5.0.jar
- lucene-core-8.5.0.jar
- lucene-sandbox-8.5.0.jar
package org.apache.lucene.codecs.compressing
- lucene-core-8.5.0.jar
- lucene-test-framework-8.5.0.jar
package org.apache.lucene.codecs.uniformsplit
- lucene-codecs-8.5.0.jar
- lucene-test-framework-8.5.0.jar
package org.apache.lucene.util
- lucene-misc-8.5.0.jar
- lucene-core-8.5.0.jar
- lucene-test-framework-8.5.0.jar
package org.apache.lucene.codecs
- lucene-core-8.5.0.jar
- lucene-test-framework-8.5.0.jar
- lucene-backward-codecs-8.5.0.jar
package org.apache.lucene.util.automaton
- lucene-core-8.5.0.jar
- lucene-test-framework-8.5.0.jar
-------
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
import java.util.zip.ZipEntry;
import java.util.zip.ZipInputStream;
public class ReadJars {
public static void main(final String[] arguments) throws IOException {
System.out.println("Find duplicate packages: <path to jar files>");
Map<String, List<String>> duplicates = new HashMap<>();
List<File> files = new ArrayList<File>();
// Find all files in sub-directory.
addFiles(files, arguments[0]);
// Find all package duplicates.
findDuplicatePackages(duplicates, files);
// Print package with duplicate packages
for (Entry<String, List<String>> entry : duplicates.entrySet()) {
System.out.println("\npackage " + entry.getKey());
for (String dup : entry.getValue()) {
System.out.println(" - " + dup);
}
}
}
private static void findDuplicatePackages(Map<String, List<String>>
duplicates, List<File> files) throws IOException {
Map<String, String> packageToJarMap = new HashMap<>();
for (File f : files) {
if (f.getAbsolutePath().endsWith(".jar")) {
String file =
f.getAbsolutePath().substring(f.getAbsolutePath().lastIndexOf('/') + 1);
ZipInputStream zip = new ZipInputStream(new
FileInputStream(f));
for (ZipEntry entry = zip.getNextEntry(); entry != null;
entry = zip.getNextEntry()) {
if (!entry.isDirectory() &&
entry.getName().endsWith(".class")) {
String className = entry.getName();
String packageName = "";
if (className.lastIndexOf('/') >= 0) {
packageName = className.substring(0,
className.lastIndexOf('/')).replace('/', '.');
}
className = className.replace('/', '.');
// Only interested in Lucene packages.
if (className.startsWith("org.apache.lucene")) {
String jar = packageToJarMap.get(packageName);
if (jar != null && !jar.equals(file)) {
List<String> duplicate =
duplicates.get(packageName);
if (duplicate == null) {
duplicate = new ArrayList<>();
duplicate.add(jar);
duplicate.add(file);
duplicates.put(packageName, duplicate);
} else {
boolean found = false;
for (String dup : duplicate) {
if (file.equals(dup)) {
found = true;
}
}
if (!found) {
duplicate.add(file);
}
}
} else {
packageToJarMap.put(packageName, file);
}
}
}
}
zip.close();
}
}
}
public static void addFiles(List<File> files, String directoryName) {
File directory = new File(directoryName);
for (File file : directory.listFiles()) {
if (file.isFile()) {
files.add(file);
} else if (file.isDirectory()) {
addFiles(files, file.getAbsolutePath());
}
}
}
}
--------
On Tue, Mar 24, 2020 at 9:26 PM Uwe Schindler <uw...@thetaphi.de> wrote:
> Hi,
>
>
>
> this is a known problem since many year and there are no plans to change
> this yet. The main reason for Java 11 support is not to introduce the
> module system, but instead use the new features of Java 11 and to get rid
> MR-JAR complications to make use of new intrinsics.
>
>
>
> Servers like Solr or Elasticsearch are shipped as an application, so the
> module system does not bring any benefit.
>
>
>
> Th general recommendation is to combine all required Lucene libraries into
> a separate JAR file during the maven / gradle build (e.g. using the Maven
> Shade plugin). Keep in mind that Lucene is also not suitable for use in
> other module systems like
>
>
>
> There is currently some preparatory things to move forward with modules,
> so although you might be able to actually compile Lucene with module system
> (by limiting to a subset of JAR files), it currently won’t work
> cross-module due to the way how it handles ServiceLoader interfaces
> (codecs, postings formats, analyzers, see
> https://issues.apache.org/jira/browse/LUCENE-9281). The only way to make
> it work at runtime is to put all of Lucene into one module.
>
>
>
> Uwe
>
>
>
> -----
>
> Uwe Schindler
>
> Achterdiek 19, D-28357 Bremen
>
> https://www.thetaphi.de
>
> eMail: uwe@thetaphi.de
>
>
>
> *From:* David Ryan <da...@livemedia.com.au>
> *Sent:* Tuesday, March 24, 2020 8:05 AM
> *To:* dev@lucene.apache.org
> *Subject:* Lucene 9.0 Java module system support
>
>
>
>
>
> Hi all,
>
>
>
> I've been investigating the use of Lucene as part of an application that
> uses the Java Module System. Initially, I used Gradle to bring in
> lucene-core, lucene-spatial and lucene-queries using version 8.5.0. This
> works with the automated module naming. However, bringing in
> lucene-queryparser (which depends on lucene-sandbox) or
> lucene-analyzers-common causes errors such as:
>
>
>
> "The package org.apache.lucene.analysis.standard is accessible from more
> than one module: lucene.analyzers.common, lucene.core"
>
>
>
> The Java module system does not handle different jar files using the same
> package name which occurs throughout multiple maven artifacts.
>
>
>
> Looking at the LUCENE issues, I found the task of moving to a minimum
> version of Java 11, however, this does not mention the ability to be
> compatible with the module system. I also checked the git repository and
> couldn't find the required changes to support the module system.
>
>
>
> https://issues.apache.org/jira/browse/LUCENE-8738
>
>
>
> I looked through the dev list recent history but could not find anything
> related. Are there any plans to support modules? Given, I saw there are a
> number of other breaking changes happening with the move to Lucene 9.0,
> would it be good to make those changes?
>
>
>
> Thanks,
>
> David.
>
>
>
Re: Lucene 9.0 Java module system support
Posted by Dawid Weiss <da...@gmail.com>.
Thanks Uwe.
On Sat, Apr 11, 2020 at 2:36 PM Uwe Schindler <uw...@thetaphi.de> wrote:
>
> Hi,
>
> If you check my plan, that's my intended way to proceed. The preparatory things like package fixes and api changes can be done before. Like the huge commit regarding service loader that was done already.
>
> Uwe
>
> Am April 11, 2020 12:27:26 PM UTC schrieb Dawid Weiss <da...@gmail.com>:
>>
>> Can we focus on moving out of ant world on master first? :) It's
>> already painful to maintain the gradle build in parallel.
>>
>> D.
>>
>> On Sat, Apr 11, 2020 at 1:14 PM Uwe Schindler <uw...@thetaphi.de> wrote:
>>>
>>>
>>> Hi,
>>>
>>>
>>>
>>> I am currently on Easter vacation, but just wanted to give some feedback:
>>>
>>> They wy you are doing it works at the moment, because you don’t have anything to do with service providers. You don’t use custom codecs (outside lucene-core.jar) and you just ignored the META-IF/services folder. Your example works, as Lucene finds everything that is needed in the core.jar, so the “legacy” service loading works (lucene-core.jar find its own services in lucene-core.jar). As soon as you would add lucene-backwards-codecs.jar, it would break unfixable, unless you copy the whole backwards codecs into core. This is completely unfixable in Lucene 8.x, as the new service-loader interface requires Java 9+.
>>>
>>>
>>>
>>> But you may not yet have noticed that I already started to do preparatory work and migrated in master (Lucene 9.0) the service provider to use the Java Runtime classes and we have thrown away our own service loader (which cannot cross module boundaries). The fix was merged not long ago: https://issues.apache.org/jira/browse/LUCENE-9281 - this cannot be backported to Java 8 / Lucene 8, as the reason for the home-made SPIClassIterator was exactly the missing features (and some classpath ordering issues in 3rd party JVMs like IBM J9). Those issues are fixed in Java 11 (also allowing to instantiate the classes on your own). When you are testing your “hack patch” with Lucene’s master branch you may succeed with also loading classes from the backwards codecs – no guarantees yet!)
>>>
>>>
>>>
>>> About StandardAnalyzer: Unfortunately I aggressively complained a while back when Mike McCandless wanted to move standard analyzer out of the analysis package into core (“for convenience”). This was a bad step, and IMHO we should revert that or completely rename the packages and everything. The problem here is: As the analysis services are only part of lucene-analyzers, we had to leave the factory classes there, but move the implementation classes in core. The package has to be the same. The only way around that is to move the analysis factory framework also to core (I would not be against that). This would include all factory base classes and the service loading stuff. Then we can move standard analyzer and some of the filters/tokenizers including their factories to core an that problem would be solved.
>>>
>>>
>>>
>>> My plan to move to modules is the following:
>>>
>>>
>>>
>>> (1) Do preparatory work:
>>>
>>> Retire SPIClassIterator (DONE).
>>> Add some preparatory issues to cleanup class hierarchy: Move Analysis SPI to core / remove StandardAnalyzer and related classes out of core back to anaysis
>>> Fix Codec API to no rely on package-private classes, so we can have a completely public API with abstract classes for codecs, so stuff in backwards-codecs does not need to have access to package private stuff in core.
>>> Cleanup sandbox to prefix all classes there with “sandbox” package and where needed remove package-private access. If it’s needed for internal access, WTF: Just move the stuff to core! We have a new version 9.0, so either retire/delete Sandbox stuff or make it part of Lucene core.
>>>
>>> (2) Wait for the Gradle Build to be finalized, because including Module stuff into the current Ant build won’t work
>>>
>>> Ant build must be retired!
>>>
>>> (3) Make Lucene real modules
>>>
>>> As we are on Java 11 already, add module-info.java everywhere
>>> Fix gradle build to create and test modules (Latest Gradle needed)
>>> Migrate all META-INF/services/* to module-info.java (before doing this, figure out of the META-INF files must stays for non-module usage, or if Java is clever enough to also look into module descriptor to find services). We may need all services at both locations (for module or classpath usage; we need a build helper to check that it’s in-line)
>>>
>>>
>>>
>>> I don’t want Lucene work with automodules in Java 11, it should be fully modularized.
>>>
>>>
>>>
>>> If you want to help and express interest: Please open an issue for the preparatory work listing the cases of same-package in different jar files. I would split this up as described above: Analysis issues with standardanalyzer, codecs pkg-private apis, sandbox.
>>>
>>>
>>>
>>> Uwe
>>> ________________________________
>>> Uwe Schindler
>>>
>>> Achterdiek 19, D-28357 Bremen
>>>
>>> https://www.thetaphi.de
>>>
>>> eMail: uwe@thetaphi.de
>>>
>>>
>>>
>>> From: David Ryan <da...@livemedia.com.au>
>>> Sent: Saturday, April 11, 2020 9:30 AM
>>> To: dev@lucene.apache.org
>>> Subject: Re: Lucene 9.0 Java module system support
>>>
>>>
>>>
>>>
>>>
>>> Thanks, I had probably missed some of what Uwe was saying in regards to the limitation of what would be possible even if the suggested changes were made. Given your points and the fact that it would take a while to have any of the changes filter into a release version, I decided to develop a patch-lucene.sh script to validate if the changes would allow me to access the required parts of Lucene. While it may not provide the full functionality provided by SPI including analysis chains and codecs, I worked out it will allow the use of the basics.
>>>
>>>
>>>
>>> The patch script below adds Automatic module names to the four Lucene libraries I needed (not strictly required, but I was interested to check if it would work). For now, any duplicate packages I identified have been moved into the core library (as identified in suggested changes). Given, I'm not using the SPI functionality, I simply deleted the standard analysers from the services list (required or the module system complains they are missing). Thankfully Gradle doesn't validate jars in the cache so the patch script only needs to be run once.
>>>
>>>
>>>
>>> Once the patch script was applied, I was able to use Ngram analysers and the query parser without issue. This will provide the essentials of what we needed for an embedded solution (location indexing, short text indexing with ngrams tokenizer and lower case filter, searching by location with nearest, bounding box and radius search, and partial text searches using the query parser). Of course, I've only scratched the surface of the APIs and right now it isn't clear if/when the SPI functionality might cause further issues or be required.
>>>
>>>
>>>
>>> So, I've validated that the suggested changes would allow basic functionality to work under the Java module system. If there's anything else I can do to help progress the suggested changes please let me know.
>>>
>>>
>>>
>>> Regards,
>>>
>>> David.
>>>
>>>
>>>
>>>
>>>
>>> --------- patch-lucene.sh -- works on osx ------
>>>
>>> mkdir -p patch-lucene/core
>>> mkdir -p patch-lucene/analyzers
>>> mkdir -p patch-lucene/sandbox
>>> mkdir -p patch-lucene/queryparser
>>> cd patch-lucene
>>>
>>> # copy the jars from the gradle cache.
>>> cp ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-core/8.5.0/3f9ea85fff4fc3f7c83869dddb9b0ef7818c0cae/lucene-core-8.5.0.jar .
>>> cp ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-analyzers-common/8.5.0/7156f2e545fd6e7faaee4781d15eb60cf5f07646/lucene-analyzers-common-8.5.0.jar .
>>> cp ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-sandbox/8.5.0/2b275921f2fd92b15b4f1a2a565467c3fa221ef9/lucene-sandbox-8.5.0.jar .
>>> cp ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queryparser/8.5.0/13c38f39b1a7d10c4749ba789fa95da5868d4885/lucene-queryparser-8.5.0.jar .
>>>
>>> # Add automatic module name to core.
>>> cd core
>>> jar -xf ../lucene-core-8.5.0.jar
>>> sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name: org.apache.lucene.core'$'\n' META-INF/MANIFEST.MF
>>>
>>> # Add automiatc module name, move standard analysis classes into core. Remove any classes in standard from service lists.
>>> cd ../analyzers
>>> jar -xf ../lucene-analyzers-common-8.5.0.jar
>>> sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name: org.apache.lucene.analyzers.common'$'\n' META-INF/MANIFEST.MF
>>> sed -i '' -e '/standard/d' META-INF/services/*
>>> mv org/apache/lucene/analysis/standard/* ../core/org/apache/lucene/analysis/standard
>>> rmdir org/apache/lucene/analysis/standard
>>> jar -cfm ../lucene-analyzers-common-8.5.0-fix.jar META-INF/MANIFEST.MF .
>>>
>>> # Add automatic module name, move search and document packages into core. move Java 9 specific search into core.
>>> cd ../sandbox
>>> jar -xf ../lucene-sandbox-8.5.0.jar
>>> sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name: org.apache.lucene.sandbox'$'\n' META-INF/MANIFEST.MF
>>> mv org/apache/lucene/search/* ../core/org/apache/lucene/search
>>> rmdir org/apache/lucene/search
>>> mv org/apache/lucene/document/* ../core/org/apache/lucene/document
>>> rmdir org/apache/lucene/document
>>> mv META-INF/versions/9/org/apache/lucene/search/* ../core/META-INF/versions/9/org/apache/lucene/search
>>> rm -rf META-INF/versions
>>> jar -cfm ../lucene-sandbox-8.5.0-fix.jar META-INF/MANIFEST.MF .
>>>
>>> # Package up core with changes.
>>> cd ../core
>>> jar -cfm ../lucene-core-8.5.0-fix.jar META-INF/MANIFEST.MF .
>>>
>>> # Add automatic module name.
>>> cd ../queryparser
>>> jar -xf ../lucene-queryparser-8.5.0.jar
>>> sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name: org.apache.lucene.queryparser'$'\n' META-INF/MANIFEST.MF
>>> jar -cfm ../lucene-queryparser-8.5.0-fix.jar META-INF/MANIFEST.MF .
>>>
>>> # Copy the fixed versions back into gradle cache.
>>> cd ..
>>> cp lucene-core-8.5.0-fix.jar ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-core/8.5.0/3f9ea85fff4fc3f7c83869dddb9b0ef7818c0cae/lucene-core-8.5.0.jar
>>> cp lucene-analyzers-common-8.5.0-fix.jar ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-analyzers-common/8.5.0/7156f2e545fd6e7faaee4781d15eb60cf5f07646/lucene-analyzers-common-8.5.0.jar
>>> cp lucene-sandbox-8.5.0-fix.jar ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-sandbox/8.5.0/2b275921f2fd92b15b4f1a2a565467c3fa221ef9/lucene-sandbox-8.5.0.jar
>>> cp lucene-queryparser-8.5.0-fix.jar ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queryparser/8.5.0/13c38f39b1a7d10c4749ba789fa95da5868d4885/lucene-queryparser-8.5.0.jar
>>> ________________________________
>>> On Sat, Apr 11, 2020 at 11:21 AM Chris Hostetter <ho...@fucit.org> wrote:
>>>
>>>
>>> : If the changes I proposed are still viewed as having too many downstream
>>> : impacts, my fallback position will be to patch the jars. This involves
>>> : using the gradle import system to get the jars from Maven, then using a
>>> : patch script to manually unzip the jars, move the offending classes into
>>> : other jars which share the same package name and rezip. So far, I've been
>>>
>>> I'm no expert here, but i trust that Uwe is, and i feel like your followup
>>> questions/suggests have still avoided his primary point about *why*
>>> Lucene/Solr hasn't attempted jigsaw modulariation...
>>>
>>> : >>> There is currently some preparatory things to move forward with modules,
>>> : >>> so although you might be able to actually compile Lucene with module system
>>> : >>> (by limiting to a subset of JAR files), it currently won’t work
>>> : >>> cross-module due to the way how it handles ServiceLoader interfaces
>>> : >>> (codecs, postings formats, analyzers, see
>>> : >>> https://issues.apache.org/jira/browse/LUCENE-9281). The only way to
>>> : >>> make it work at runtime is to put all of Lucene into one module.
>>>
>>> ...so, IIUC, even if you patch the *current* jars, any Lucene code you use
>>> that depends on SPI (like analysis chains, codecs, etc...) isn't going to
>>> work unless follow Uwe's primary suggestion for folks who care about
>>> modules...
>>>
>>> : >>> Th general recommendation is to combine all required Lucene libraries
>>> : >>> into a separate JAR file during the maven / gradle build (e.g. using the
>>> : >>> Maven Shade plugin). Keep in mind that Lucene is also not suitable for use
>>>
>>>
>>> -Hoss
>>> http://www.lucidworks.com/
>>> ________________________________
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>> ________________________________
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
> --
> Uwe Schindler
> Achterdiek 19, 28357 Bremen
> https://www.thetaphi.de
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Lucene 9.0 Java module system support
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
If you check my plan, that's my intended way to proceed. The preparatory things like package fixes and api changes can be done before. Like the huge commit regarding service loader that was done already.
Uwe
Am April 11, 2020 12:27:26 PM UTC schrieb Dawid Weiss <da...@gmail.com>:
>Can we focus on moving out of ant world on master first? :) It's
>already painful to maintain the gradle build in parallel.
>
>D.
>
>On Sat, Apr 11, 2020 at 1:14 PM Uwe Schindler <uw...@thetaphi.de> wrote:
>>
>> Hi,
>>
>>
>>
>> I am currently on Easter vacation, but just wanted to give some
>feedback:
>>
>> They wy you are doing it works at the moment, because you don’t have
>anything to do with service providers. You don’t use custom codecs
>(outside lucene-core.jar) and you just ignored the META-IF/services
>folder. Your example works, as Lucene finds everything that is needed
>in the core.jar, so the “legacy” service loading works (lucene-core.jar
>find its own services in lucene-core.jar). As soon as you would add
>lucene-backwards-codecs.jar, it would break unfixable, unless you copy
>the whole backwards codecs into core. This is completely unfixable in
>Lucene 8.x, as the new service-loader interface requires Java 9+.
>>
>>
>>
>> But you may not yet have noticed that I already started to do
>preparatory work and migrated in master (Lucene 9.0) the service
>provider to use the Java Runtime classes and we have thrown away our
>own service loader (which cannot cross module boundaries). The fix was
>merged not long ago: https://issues.apache.org/jira/browse/LUCENE-9281
>- this cannot be backported to Java 8 / Lucene 8, as the reason for the
>home-made SPIClassIterator was exactly the missing features (and some
>classpath ordering issues in 3rd party JVMs like IBM J9). Those issues
>are fixed in Java 11 (also allowing to instantiate the classes on your
>own). When you are testing your “hack patch” with Lucene’s master
>branch you may succeed with also loading classes from the backwards
>codecs – no guarantees yet!)
>>
>>
>>
>> About StandardAnalyzer: Unfortunately I aggressively complained a
>while back when Mike McCandless wanted to move standard analyzer out of
>the analysis package into core (“for convenience”). This was a bad
>step, and IMHO we should revert that or completely rename the packages
>and everything. The problem here is: As the analysis services are only
>part of lucene-analyzers, we had to leave the factory classes there,
>but move the implementation classes in core. The package has to be the
>same. The only way around that is to move the analysis factory
>framework also to core (I would not be against that). This would
>include all factory base classes and the service loading stuff. Then we
>can move standard analyzer and some of the filters/tokenizers including
>their factories to core an that problem would be solved.
>>
>>
>>
>> My plan to move to modules is the following:
>>
>>
>>
>> (1) Do preparatory work:
>>
>> Retire SPIClassIterator (DONE).
>> Add some preparatory issues to cleanup class hierarchy: Move Analysis
>SPI to core / remove StandardAnalyzer and related classes out of core
>back to anaysis
>> Fix Codec API to no rely on package-private classes, so we can have a
>completely public API with abstract classes for codecs, so stuff in
>backwards-codecs does not need to have access to package private stuff
>in core.
>> Cleanup sandbox to prefix all classes there with “sandbox” package
>and where needed remove package-private access. If it’s needed for
>internal access, WTF: Just move the stuff to core! We have a new
>version 9.0, so either retire/delete Sandbox stuff or make it part of
>Lucene core.
>>
>> (2) Wait for the Gradle Build to be finalized, because including
>Module stuff into the current Ant build won’t work
>>
>> Ant build must be retired!
>>
>> (3) Make Lucene real modules
>>
>> As we are on Java 11 already, add module-info.java everywhere
>> Fix gradle build to create and test modules (Latest Gradle needed)
>> Migrate all META-INF/services/* to module-info.java (before doing
>this, figure out of the META-INF files must stays for non-module usage,
>or if Java is clever enough to also look into module descriptor to find
>services). We may need all services at both locations (for module or
>classpath usage; we need a build helper to check that it’s in-line)
>>
>>
>>
>> I don’t want Lucene work with automodules in Java 11, it should be
>fully modularized.
>>
>>
>>
>> If you want to help and express interest: Please open an issue for
>the preparatory work listing the cases of same-package in different jar
>files. I would split this up as described above: Analysis issues with
>standardanalyzer, codecs pkg-private apis, sandbox.
>>
>>
>>
>> Uwe
>>
>>
>>
>> -----
>>
>> Uwe Schindler
>>
>> Achterdiek 19, D-28357 Bremen
>>
>> https://www.thetaphi.de
>>
>> eMail: uwe@thetaphi.de
>>
>>
>>
>> From: David Ryan <da...@livemedia.com.au>
>> Sent: Saturday, April 11, 2020 9:30 AM
>> To: dev@lucene.apache.org
>> Subject: Re: Lucene 9.0 Java module system support
>>
>>
>>
>>
>>
>> Thanks, I had probably missed some of what Uwe was saying in regards
>to the limitation of what would be possible even if the suggested
>changes were made. Given your points and the fact that it would take a
>while to have any of the changes filter into a release version, I
>decided to develop a patch-lucene.sh script to validate if the changes
>would allow me to access the required parts of Lucene. While it may
>not provide the full functionality provided by SPI including analysis
>chains and codecs, I worked out it will allow the use of the basics.
>>
>>
>>
>> The patch script below adds Automatic module names to the four Lucene
>libraries I needed (not strictly required, but I was interested to
>check if it would work). For now, any duplicate packages I identified
>have been moved into the core library (as identified in suggested
>changes). Given, I'm not using the SPI functionality, I simply deleted
>the standard analysers from the services list (required or the module
>system complains they are missing). Thankfully Gradle doesn't validate
>jars in the cache so the patch script only needs to be run once.
>>
>>
>>
>> Once the patch script was applied, I was able to use Ngram analysers
>and the query parser without issue. This will provide the essentials of
>what we needed for an embedded solution (location indexing, short text
>indexing with ngrams tokenizer and lower case filter, searching by
>location with nearest, bounding box and radius search, and partial text
>searches using the query parser). Of course, I've only scratched the
>surface of the APIs and right now it isn't clear if/when the SPI
>functionality might cause further issues or be required.
>>
>>
>>
>> So, I've validated that the suggested changes would allow basic
>functionality to work under the Java module system. If there's anything
>else I can do to help progress the suggested changes please let me
>know.
>>
>>
>>
>> Regards,
>>
>> David.
>>
>>
>>
>>
>>
>> --------- patch-lucene.sh -- works on osx ------
>>
>> mkdir -p patch-lucene/core
>> mkdir -p patch-lucene/analyzers
>> mkdir -p patch-lucene/sandbox
>> mkdir -p patch-lucene/queryparser
>> cd patch-lucene
>>
>> # copy the jars from the gradle cache.
>> cp
>~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-core/8.5.0/3f9ea85fff4fc3f7c83869dddb9b0ef7818c0cae/lucene-core-8.5.0.jar
>.
>> cp
>~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-analyzers-common/8.5.0/7156f2e545fd6e7faaee4781d15eb60cf5f07646/lucene-analyzers-common-8.5.0.jar
>.
>> cp
>~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-sandbox/8.5.0/2b275921f2fd92b15b4f1a2a565467c3fa221ef9/lucene-sandbox-8.5.0.jar
>.
>> cp
>~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queryparser/8.5.0/13c38f39b1a7d10c4749ba789fa95da5868d4885/lucene-queryparser-8.5.0.jar
>.
>>
>> # Add automatic module name to core.
>> cd core
>> jar -xf ../lucene-core-8.5.0.jar
>> sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name:
>org.apache.lucene.core'$'\n' META-INF/MANIFEST.MF
>>
>> # Add automiatc module name, move standard analysis classes into
>core. Remove any classes in standard from service lists.
>> cd ../analyzers
>> jar -xf ../lucene-analyzers-common-8.5.0.jar
>> sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name:
>org.apache.lucene.analyzers.common'$'\n' META-INF/MANIFEST.MF
>> sed -i '' -e '/standard/d' META-INF/services/*
>> mv org/apache/lucene/analysis/standard/*
>../core/org/apache/lucene/analysis/standard
>> rmdir org/apache/lucene/analysis/standard
>> jar -cfm ../lucene-analyzers-common-8.5.0-fix.jar
>META-INF/MANIFEST.MF .
>>
>> # Add automatic module name, move search and document packages into
>core. move Java 9 specific search into core.
>> cd ../sandbox
>> jar -xf ../lucene-sandbox-8.5.0.jar
>> sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name:
>org.apache.lucene.sandbox'$'\n' META-INF/MANIFEST.MF
>> mv org/apache/lucene/search/* ../core/org/apache/lucene/search
>> rmdir org/apache/lucene/search
>> mv org/apache/lucene/document/* ../core/org/apache/lucene/document
>> rmdir org/apache/lucene/document
>> mv META-INF/versions/9/org/apache/lucene/search/*
>../core/META-INF/versions/9/org/apache/lucene/search
>> rm -rf META-INF/versions
>> jar -cfm ../lucene-sandbox-8.5.0-fix.jar META-INF/MANIFEST.MF .
>>
>> # Package up core with changes.
>> cd ../core
>> jar -cfm ../lucene-core-8.5.0-fix.jar META-INF/MANIFEST.MF .
>>
>> # Add automatic module name.
>> cd ../queryparser
>> jar -xf ../lucene-queryparser-8.5.0.jar
>> sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name:
>org.apache.lucene.queryparser'$'\n' META-INF/MANIFEST.MF
>> jar -cfm ../lucene-queryparser-8.5.0-fix.jar META-INF/MANIFEST.MF .
>>
>> # Copy the fixed versions back into gradle cache.
>> cd ..
>> cp lucene-core-8.5.0-fix.jar
>~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-core/8.5.0/3f9ea85fff4fc3f7c83869dddb9b0ef7818c0cae/lucene-core-8.5.0.jar
>> cp lucene-analyzers-common-8.5.0-fix.jar
>~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-analyzers-common/8.5.0/7156f2e545fd6e7faaee4781d15eb60cf5f07646/lucene-analyzers-common-8.5.0.jar
>> cp lucene-sandbox-8.5.0-fix.jar
>~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-sandbox/8.5.0/2b275921f2fd92b15b4f1a2a565467c3fa221ef9/lucene-sandbox-8.5.0.jar
>> cp lucene-queryparser-8.5.0-fix.jar
>~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queryparser/8.5.0/13c38f39b1a7d10c4749ba789fa95da5868d4885/lucene-queryparser-8.5.0.jar
>>
>> ------------
>>
>>
>>
>> On Sat, Apr 11, 2020 at 11:21 AM Chris Hostetter
><ho...@fucit.org> wrote:
>>
>>
>> : If the changes I proposed are still viewed as having too many
>downstream
>> : impacts, my fallback position will be to patch the jars. This
>involves
>> : using the gradle import system to get the jars from Maven, then
>using a
>> : patch script to manually unzip the jars, move the offending classes
>into
>> : other jars which share the same package name and rezip. So far,
>I've been
>>
>> I'm no expert here, but i trust that Uwe is, and i feel like your
>followup
>> questions/suggests have still avoided his primary point about *why*
>> Lucene/Solr hasn't attempted jigsaw modulariation...
>>
>> : >>> There is currently some preparatory things to move forward with
>modules,
>> : >>> so although you might be able to actually compile Lucene with
>module system
>> : >>> (by limiting to a subset of JAR files), it currently won’t work
>> : >>> cross-module due to the way how it handles ServiceLoader
>interfaces
>> : >>> (codecs, postings formats, analyzers, see
>> : >>> https://issues.apache.org/jira/browse/LUCENE-9281). The only
>way to
>> : >>> make it work at runtime is to put all of Lucene into one
>module.
>>
>> ...so, IIUC, even if you patch the *current* jars, any Lucene code
>you use
>> that depends on SPI (like analysis chains, codecs, etc...) isn't
>going to
>> work unless follow Uwe's primary suggestion for folks who care about
>> modules...
>>
>> : >>> Th general recommendation is to combine all required Lucene
>libraries
>> : >>> into a separate JAR file during the maven / gradle build (e.g.
>using the
>> : >>> Maven Shade plugin). Keep in mind that Lucene is also not
>suitable for use
>>
>>
>> -Hoss
>> http://www.lucidworks.com/
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>For additional commands, e-mail: dev-help@lucene.apache.org
--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de
Re: Lucene 9.0 Java module system support
Posted by Dawid Weiss <da...@gmail.com>.
Can we focus on moving out of ant world on master first? :) It's
already painful to maintain the gradle build in parallel.
D.
On Sat, Apr 11, 2020 at 1:14 PM Uwe Schindler <uw...@thetaphi.de> wrote:
>
> Hi,
>
>
>
> I am currently on Easter vacation, but just wanted to give some feedback:
>
> They wy you are doing it works at the moment, because you don’t have anything to do with service providers. You don’t use custom codecs (outside lucene-core.jar) and you just ignored the META-IF/services folder. Your example works, as Lucene finds everything that is needed in the core.jar, so the “legacy” service loading works (lucene-core.jar find its own services in lucene-core.jar). As soon as you would add lucene-backwards-codecs.jar, it would break unfixable, unless you copy the whole backwards codecs into core. This is completely unfixable in Lucene 8.x, as the new service-loader interface requires Java 9+.
>
>
>
> But you may not yet have noticed that I already started to do preparatory work and migrated in master (Lucene 9.0) the service provider to use the Java Runtime classes and we have thrown away our own service loader (which cannot cross module boundaries). The fix was merged not long ago: https://issues.apache.org/jira/browse/LUCENE-9281 - this cannot be backported to Java 8 / Lucene 8, as the reason for the home-made SPIClassIterator was exactly the missing features (and some classpath ordering issues in 3rd party JVMs like IBM J9). Those issues are fixed in Java 11 (also allowing to instantiate the classes on your own). When you are testing your “hack patch” with Lucene’s master branch you may succeed with also loading classes from the backwards codecs – no guarantees yet!)
>
>
>
> About StandardAnalyzer: Unfortunately I aggressively complained a while back when Mike McCandless wanted to move standard analyzer out of the analysis package into core (“for convenience”). This was a bad step, and IMHO we should revert that or completely rename the packages and everything. The problem here is: As the analysis services are only part of lucene-analyzers, we had to leave the factory classes there, but move the implementation classes in core. The package has to be the same. The only way around that is to move the analysis factory framework also to core (I would not be against that). This would include all factory base classes and the service loading stuff. Then we can move standard analyzer and some of the filters/tokenizers including their factories to core an that problem would be solved.
>
>
>
> My plan to move to modules is the following:
>
>
>
> (1) Do preparatory work:
>
> Retire SPIClassIterator (DONE).
> Add some preparatory issues to cleanup class hierarchy: Move Analysis SPI to core / remove StandardAnalyzer and related classes out of core back to anaysis
> Fix Codec API to no rely on package-private classes, so we can have a completely public API with abstract classes for codecs, so stuff in backwards-codecs does not need to have access to package private stuff in core.
> Cleanup sandbox to prefix all classes there with “sandbox” package and where needed remove package-private access. If it’s needed for internal access, WTF: Just move the stuff to core! We have a new version 9.0, so either retire/delete Sandbox stuff or make it part of Lucene core.
>
> (2) Wait for the Gradle Build to be finalized, because including Module stuff into the current Ant build won’t work
>
> Ant build must be retired!
>
> (3) Make Lucene real modules
>
> As we are on Java 11 already, add module-info.java everywhere
> Fix gradle build to create and test modules (Latest Gradle needed)
> Migrate all META-INF/services/* to module-info.java (before doing this, figure out of the META-INF files must stays for non-module usage, or if Java is clever enough to also look into module descriptor to find services). We may need all services at both locations (for module or classpath usage; we need a build helper to check that it’s in-line)
>
>
>
> I don’t want Lucene work with automodules in Java 11, it should be fully modularized.
>
>
>
> If you want to help and express interest: Please open an issue for the preparatory work listing the cases of same-package in different jar files. I would split this up as described above: Analysis issues with standardanalyzer, codecs pkg-private apis, sandbox.
>
>
>
> Uwe
>
>
>
> -----
>
> Uwe Schindler
>
> Achterdiek 19, D-28357 Bremen
>
> https://www.thetaphi.de
>
> eMail: uwe@thetaphi.de
>
>
>
> From: David Ryan <da...@livemedia.com.au>
> Sent: Saturday, April 11, 2020 9:30 AM
> To: dev@lucene.apache.org
> Subject: Re: Lucene 9.0 Java module system support
>
>
>
>
>
> Thanks, I had probably missed some of what Uwe was saying in regards to the limitation of what would be possible even if the suggested changes were made. Given your points and the fact that it would take a while to have any of the changes filter into a release version, I decided to develop a patch-lucene.sh script to validate if the changes would allow me to access the required parts of Lucene. While it may not provide the full functionality provided by SPI including analysis chains and codecs, I worked out it will allow the use of the basics.
>
>
>
> The patch script below adds Automatic module names to the four Lucene libraries I needed (not strictly required, but I was interested to check if it would work). For now, any duplicate packages I identified have been moved into the core library (as identified in suggested changes). Given, I'm not using the SPI functionality, I simply deleted the standard analysers from the services list (required or the module system complains they are missing). Thankfully Gradle doesn't validate jars in the cache so the patch script only needs to be run once.
>
>
>
> Once the patch script was applied, I was able to use Ngram analysers and the query parser without issue. This will provide the essentials of what we needed for an embedded solution (location indexing, short text indexing with ngrams tokenizer and lower case filter, searching by location with nearest, bounding box and radius search, and partial text searches using the query parser). Of course, I've only scratched the surface of the APIs and right now it isn't clear if/when the SPI functionality might cause further issues or be required.
>
>
>
> So, I've validated that the suggested changes would allow basic functionality to work under the Java module system. If there's anything else I can do to help progress the suggested changes please let me know.
>
>
>
> Regards,
>
> David.
>
>
>
>
>
> --------- patch-lucene.sh -- works on osx ------
>
> mkdir -p patch-lucene/core
> mkdir -p patch-lucene/analyzers
> mkdir -p patch-lucene/sandbox
> mkdir -p patch-lucene/queryparser
> cd patch-lucene
>
> # copy the jars from the gradle cache.
> cp ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-core/8.5.0/3f9ea85fff4fc3f7c83869dddb9b0ef7818c0cae/lucene-core-8.5.0.jar .
> cp ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-analyzers-common/8.5.0/7156f2e545fd6e7faaee4781d15eb60cf5f07646/lucene-analyzers-common-8.5.0.jar .
> cp ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-sandbox/8.5.0/2b275921f2fd92b15b4f1a2a565467c3fa221ef9/lucene-sandbox-8.5.0.jar .
> cp ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queryparser/8.5.0/13c38f39b1a7d10c4749ba789fa95da5868d4885/lucene-queryparser-8.5.0.jar .
>
> # Add automatic module name to core.
> cd core
> jar -xf ../lucene-core-8.5.0.jar
> sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name: org.apache.lucene.core'$'\n' META-INF/MANIFEST.MF
>
> # Add automiatc module name, move standard analysis classes into core. Remove any classes in standard from service lists.
> cd ../analyzers
> jar -xf ../lucene-analyzers-common-8.5.0.jar
> sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name: org.apache.lucene.analyzers.common'$'\n' META-INF/MANIFEST.MF
> sed -i '' -e '/standard/d' META-INF/services/*
> mv org/apache/lucene/analysis/standard/* ../core/org/apache/lucene/analysis/standard
> rmdir org/apache/lucene/analysis/standard
> jar -cfm ../lucene-analyzers-common-8.5.0-fix.jar META-INF/MANIFEST.MF .
>
> # Add automatic module name, move search and document packages into core. move Java 9 specific search into core.
> cd ../sandbox
> jar -xf ../lucene-sandbox-8.5.0.jar
> sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name: org.apache.lucene.sandbox'$'\n' META-INF/MANIFEST.MF
> mv org/apache/lucene/search/* ../core/org/apache/lucene/search
> rmdir org/apache/lucene/search
> mv org/apache/lucene/document/* ../core/org/apache/lucene/document
> rmdir org/apache/lucene/document
> mv META-INF/versions/9/org/apache/lucene/search/* ../core/META-INF/versions/9/org/apache/lucene/search
> rm -rf META-INF/versions
> jar -cfm ../lucene-sandbox-8.5.0-fix.jar META-INF/MANIFEST.MF .
>
> # Package up core with changes.
> cd ../core
> jar -cfm ../lucene-core-8.5.0-fix.jar META-INF/MANIFEST.MF .
>
> # Add automatic module name.
> cd ../queryparser
> jar -xf ../lucene-queryparser-8.5.0.jar
> sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name: org.apache.lucene.queryparser'$'\n' META-INF/MANIFEST.MF
> jar -cfm ../lucene-queryparser-8.5.0-fix.jar META-INF/MANIFEST.MF .
>
> # Copy the fixed versions back into gradle cache.
> cd ..
> cp lucene-core-8.5.0-fix.jar ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-core/8.5.0/3f9ea85fff4fc3f7c83869dddb9b0ef7818c0cae/lucene-core-8.5.0.jar
> cp lucene-analyzers-common-8.5.0-fix.jar ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-analyzers-common/8.5.0/7156f2e545fd6e7faaee4781d15eb60cf5f07646/lucene-analyzers-common-8.5.0.jar
> cp lucene-sandbox-8.5.0-fix.jar ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-sandbox/8.5.0/2b275921f2fd92b15b4f1a2a565467c3fa221ef9/lucene-sandbox-8.5.0.jar
> cp lucene-queryparser-8.5.0-fix.jar ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queryparser/8.5.0/13c38f39b1a7d10c4749ba789fa95da5868d4885/lucene-queryparser-8.5.0.jar
>
> ------------
>
>
>
> On Sat, Apr 11, 2020 at 11:21 AM Chris Hostetter <ho...@fucit.org> wrote:
>
>
> : If the changes I proposed are still viewed as having too many downstream
> : impacts, my fallback position will be to patch the jars. This involves
> : using the gradle import system to get the jars from Maven, then using a
> : patch script to manually unzip the jars, move the offending classes into
> : other jars which share the same package name and rezip. So far, I've been
>
> I'm no expert here, but i trust that Uwe is, and i feel like your followup
> questions/suggests have still avoided his primary point about *why*
> Lucene/Solr hasn't attempted jigsaw modulariation...
>
> : >>> There is currently some preparatory things to move forward with modules,
> : >>> so although you might be able to actually compile Lucene with module system
> : >>> (by limiting to a subset of JAR files), it currently won’t work
> : >>> cross-module due to the way how it handles ServiceLoader interfaces
> : >>> (codecs, postings formats, analyzers, see
> : >>> https://issues.apache.org/jira/browse/LUCENE-9281). The only way to
> : >>> make it work at runtime is to put all of Lucene into one module.
>
> ...so, IIUC, even if you patch the *current* jars, any Lucene code you use
> that depends on SPI (like analysis chains, codecs, etc...) isn't going to
> work unless follow Uwe's primary suggestion for folks who care about
> modules...
>
> : >>> Th general recommendation is to combine all required Lucene libraries
> : >>> into a separate JAR file during the maven / gradle build (e.g. using the
> : >>> Maven Shade plugin). Keep in mind that Lucene is also not suitable for use
>
>
> -Hoss
> http://www.lucidworks.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Lucene 9.0 Java module system support
Posted by Tomoko Uchida <to...@gmail.com>.
Hi,
I left comments for your questions about development setup on LUCENE-9317.
Please see them.
Tomoko
2020年4月12日(日) 14:16 David Ryan <da...@livemedia.com.au>:
>
> Hi Uwe,
>
> Thanks for taking the time out of your Easter break to respond. I've
> created the following Jira tickets for preparatory tasks:
>
> LUCENE-9317 <https://issues.apache.org/jira/browse/LUCENE-9317> Resolve
> package name conflicts for StandardAnalyzer to allow Java module system
> support
> LUCENE-9318 <https://issues.apache.org/jira/browse/LUCENE-9318> Fix Codec
> API to not rely on package-private classes as part of changes to support
> java module system
> LUCENE-9319 <https://issues.apache.org/jira/browse/LUCENE-9319> Clean up
> Sandbox project by retiring/delete functionality or move it to Lucene core
>
> I've also created the following for adding module-info.java:
>
> LUCENE-9320 <https://issues.apache.org/jira/browse/LUCENE-9320> Enable
> support the Java module system by adding module-info to projects
>
> I've checked out the master branch and starting to look at setting up the
> development environment using Gradle to see if I can assist with a few
> items. I notice Gradle hasn't been set up for Eclipse yet (maybe something
> I can help with). My first roadblock to getting setup is understanding the
> cyclic dependencies between core, codecs and the test-framework. Given IRC
> is not really used from what I can see, and the slack channel is private to
> people with @apache.org emails. Is there any good place to get basic help
> on getting started with the codebase without spamming this email list?
>
> Thanks,
> David.
>
> On Sat, Apr 11, 2020 at 9:14 PM Uwe Schindler <uw...@thetaphi.de> wrote:
>
>> Hi,
>>
>>
>>
>> I am currently on Easter vacation, but just wanted to give some feedback:
>>
>> They wy you are doing it works at the moment, because you don’t have
>> anything to do with service providers. You don’t use custom codecs (outside
>> lucene-core.jar) and you just ignored the META-IF/services folder. Your
>> example works, as Lucene finds everything that is needed in the core.jar,
>> so the “legacy” service loading works (lucene-core.jar find its own
>> services in lucene-core.jar). As soon as you would add
>> lucene-backwards-codecs.jar, it would break unfixable, unless you copy the
>> whole backwards codecs into core. This is completely unfixable in Lucene
>> 8.x, as the new service-loader interface requires Java 9+.
>>
>>
>>
>> But you may not yet have noticed that I already started to do preparatory
>> work and migrated in master (Lucene 9.0) the service provider to use the
>> Java Runtime classes and we have thrown away our own service loader (which
>> cannot cross module boundaries). The fix was merged not long ago:
>> https://issues.apache.org/jira/browse/LUCENE-9281 - this cannot be
>> backported to Java 8 / Lucene 8, as the reason for the home-made
>> SPIClassIterator was exactly the missing features (and some classpath
>> ordering issues in 3rd party JVMs like IBM J9). Those issues are fixed
>> in Java 11 (also allowing to instantiate the classes on your own). When you
>> are testing your “hack patch” with Lucene’s master branch you may succeed
>> with also loading classes from the backwards codecs – no guarantees yet!)
>>
>>
>>
>> About StandardAnalyzer: Unfortunately I aggressively complained a while
>> back when Mike McCandless wanted to move standard analyzer out of the
>> analysis package into core (“for convenience”). This was a bad step, and
>> IMHO we should revert that or completely rename the packages and
>> everything. The problem here is: As the analysis services are only part of
>> lucene-analyzers, we had to leave the factory classes there, but move the
>> implementation classes in core. The package has to be the same. The only
>> way around that is to move the analysis factory framework also to core (I
>> would not be against that). This would include all factory base classes and
>> the service loading stuff. Then we can move standard analyzer and some of
>> the filters/tokenizers including their factories to core an that problem
>> would be solved.
>>
>>
>>
>> My plan to move to modules is the following:
>>
>>
>>
>> (1) Do preparatory work:
>>
>> - Retire SPIClassIterator (DONE).
>> - Add some preparatory issues to cleanup class hierarchy: Move
>> Analysis SPI to core / remove StandardAnalyzer and related classes out of
>> core back to anaysis
>> - Fix Codec API to no rely on package-private classes, so we can have
>> a completely public API with abstract classes for codecs, so stuff in
>> backwards-codecs does not need to have access to package private stuff in
>> core.
>> - Cleanup sandbox to prefix all classes there with “sandbox” package
>> and where needed remove package-private access. If it’s needed for internal
>> access, WTF: Just move the stuff to core! We have a new version 9.0, so
>> either retire/delete Sandbox stuff or make it part of Lucene core.
>>
>> (2) Wait for the Gradle Build to be finalized, because including Module
>> stuff into the current Ant build won’t work
>>
>> - Ant build must be retired!
>>
>> (3) Make Lucene real modules
>>
>> - As we are on Java 11 already, add module-info.java everywhere
>> - Fix gradle build to create and test modules (Latest Gradle needed)
>> - Migrate all META-INF/services/* to module-info.java (before doing
>> this, figure out of the META-INF files must stays for non-module usage, or
>> if Java is clever enough to also look into module descriptor to find
>> services). We may need all services at both locations (for module or
>> classpath usage; we need a build helper to check that it’s in-line)
>>
>>
>>
>> I don’t want Lucene work with automodules in Java 11, it should be fully
>> modularized.
>>
>>
>>
>> If you want to help and express interest: Please open an issue for the
>> preparatory work listing the cases of same-package in different jar files.
>> I would split this up as described above: Analysis issues with
>> standardanalyzer, codecs pkg-private apis, sandbox.
>>
>>
>>
>> Uwe
>>
>>
>>
>> -----
>>
>> Uwe Schindler
>>
>> Achterdiek 19, D-28357 Bremen
>>
>> https://www.thetaphi.de
>>
>> eMail: uwe@thetaphi.de
>>
>>
>>
>> *From:* David Ryan <da...@livemedia.com.au>
>> *Sent:* Saturday, April 11, 2020 9:30 AM
>> *To:* dev@lucene.apache.org
>> *Subject:* Re: Lucene 9.0 Java module system support
>>
>>
>>
>>
>>
>> Thanks, I had probably missed some of what Uwe was saying in regards to
>> the limitation of what would be possible even if the suggested changes were
>> made. Given your points and the fact that it would take a while to have any
>> of the changes filter into a release version, I decided to develop a
>> patch-lucene.sh script to validate if the changes would allow me to access
>> the required parts of Lucene. While it may not provide the full
>> functionality provided by SPI including analysis chains and codecs, I
>> worked out it will allow the use of the basics.
>>
>>
>>
>> The patch script below adds Automatic module names to the four Lucene
>> libraries I needed (not strictly required, but I was interested to check if
>> it would work). For now, any duplicate packages I identified have been
>> moved into the core library (as identified in suggested changes). Given,
>> I'm not using the SPI functionality, I simply deleted the standard
>> analysers from the services list (required or the module system complains
>> they are missing). Thankfully Gradle doesn't validate jars in the cache so
>> the patch script only needs to be run once.
>>
>>
>>
>> Once the patch script was applied, I was able to use Ngram analysers and
>> the query parser without issue. This will provide the essentials of what we
>> needed for an embedded solution (location indexing, short text indexing
>> with ngrams tokenizer and lower case filter, searching by location with
>> nearest, bounding box and radius search, and partial text searches using
>> the query parser). Of course, I've only scratched the surface of the APIs
>> and right now it isn't clear if/when the SPI functionality might cause
>> further issues or be required.
>>
>>
>>
>> So, I've validated that the suggested changes would allow basic
>> functionality to work under the Java module system. If there's anything
>> else I can do to help progress the suggested changes please let me know.
>>
>>
>>
>> Regards,
>>
>> David.
>>
>>
>>
>>
>>
>> --------- patch-lucene.sh -- works on osx ------
>>
>> mkdir -p patch-lucene/core
>> mkdir -p patch-lucene/analyzers
>> mkdir -p patch-lucene/sandbox
>> mkdir -p patch-lucene/queryparser
>> cd patch-lucene
>>
>> # copy the jars from the gradle cache.
>> cp
>> ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-core/8.5.0/3f9ea85fff4fc3f7c83869dddb9b0ef7818c0cae/lucene-core-8.5.0.jar
>> .
>> cp
>> ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-analyzers-common/8.5.0/7156f2e545fd6e7faaee4781d15eb60cf5f07646/lucene-analyzers-common-8.5.0.jar
>> .
>> cp
>> ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-sandbox/8.5.0/2b275921f2fd92b15b4f1a2a565467c3fa221ef9/lucene-sandbox-8.5.0.jar
>> .
>> cp
>> ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queryparser/8.5.0/13c38f39b1a7d10c4749ba789fa95da5868d4885/lucene-queryparser-8.5.0.jar
>> .
>>
>> # Add automatic module name to core.
>> cd core
>> jar -xf ../lucene-core-8.5.0.jar
>> sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name:
>> org.apache.lucene.core'$'\n' META-INF/MANIFEST.MF
>>
>> # Add automiatc module name, move standard analysis classes into core.
>> Remove any classes in standard from service lists.
>> cd ../analyzers
>> jar -xf ../lucene-analyzers-common-8.5.0.jar
>> sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name:
>> org.apache.lucene.analyzers.common'$'\n' META-INF/MANIFEST.MF
>> sed -i '' -e '/standard/d' META-INF/services/*
>> mv org/apache/lucene/analysis/standard/*
>> ../core/org/apache/lucene/analysis/standard
>> rmdir org/apache/lucene/analysis/standard
>> jar -cfm ../lucene-analyzers-common-8.5.0-fix.jar META-INF/MANIFEST.MF .
>>
>> # Add automatic module name, move search and document packages into core.
>> move Java 9 specific search into core.
>> cd ../sandbox
>> jar -xf ../lucene-sandbox-8.5.0.jar
>> sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name:
>> org.apache.lucene.sandbox'$'\n' META-INF/MANIFEST.MF
>> mv org/apache/lucene/search/* ../core/org/apache/lucene/search
>> rmdir org/apache/lucene/search
>> mv org/apache/lucene/document/* ../core/org/apache/lucene/document
>> rmdir org/apache/lucene/document
>> mv META-INF/versions/9/org/apache/lucene/search/*
>> ../core/META-INF/versions/9/org/apache/lucene/search
>> rm -rf META-INF/versions
>> jar -cfm ../lucene-sandbox-8.5.0-fix.jar META-INF/MANIFEST.MF .
>>
>> # Package up core with changes.
>> cd ../core
>> jar -cfm ../lucene-core-8.5.0-fix.jar META-INF/MANIFEST.MF .
>>
>> # Add automatic module name.
>> cd ../queryparser
>> jar -xf ../lucene-queryparser-8.5.0.jar
>> sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name:
>> org.apache.lucene.queryparser'$'\n' META-INF/MANIFEST.MF
>> jar -cfm ../lucene-queryparser-8.5.0-fix.jar META-INF/MANIFEST.MF .
>>
>> # Copy the fixed versions back into gradle cache.
>> cd ..
>> cp lucene-core-8.5.0-fix.jar
>> ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-core/8.5.0/3f9ea85fff4fc3f7c83869dddb9b0ef7818c0cae/lucene-core-8.5.0.jar
>> cp lucene-analyzers-common-8.5.0-fix.jar
>> ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-analyzers-common/8.5.0/7156f2e545fd6e7faaee4781d15eb60cf5f07646/lucene-analyzers-common-8.5.0.jar
>> cp lucene-sandbox-8.5.0-fix.jar
>> ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-sandbox/8.5.0/2b275921f2fd92b15b4f1a2a565467c3fa221ef9/lucene-sandbox-8.5.0.jar
>> cp lucene-queryparser-8.5.0-fix.jar
>> ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queryparser/8.5.0/13c38f39b1a7d10c4749ba789fa95da5868d4885/lucene-queryparser-8.5.0.jar
>>
>> ------------
>>
>>
>>
>> On Sat, Apr 11, 2020 at 11:21 AM Chris Hostetter <
>> hossman_lucene@fucit.org> wrote:
>>
>>
>> : If the changes I proposed are still viewed as having too many downstream
>> : impacts, my fallback position will be to patch the jars. This involves
>> : using the gradle import system to get the jars from Maven, then using a
>> : patch script to manually unzip the jars, move the offending classes into
>> : other jars which share the same package name and rezip. So far, I've
>> been
>>
>> I'm no expert here, but i trust that Uwe is, and i feel like your
>> followup
>> questions/suggests have still avoided his primary point about *why*
>> Lucene/Solr hasn't attempted jigsaw modulariation...
>>
>> : >>> There is currently some preparatory things to move forward with
>> modules,
>> : >>> so although you might be able to actually compile Lucene with
>> module system
>> : >>> (by limiting to a subset of JAR files), it currently won’t work
>> : >>> cross-module due to the way how it handles ServiceLoader interfaces
>> : >>> (codecs, postings formats, analyzers, see
>> : >>> https://issues.apache.org/jira/browse/LUCENE-9281). The only way to
>> : >>> make it work at runtime is to put all of Lucene into one module.
>>
>> ...so, IIUC, even if you patch the *current* jars, any Lucene code you
>> use
>> that depends on SPI (like analysis chains, codecs, etc...) isn't going to
>> work unless follow Uwe's primary suggestion for folks who care about
>> modules...
>>
>> : >>> Th general recommendation is to combine all required Lucene
>> libraries
>> : >>> into a separate JAR file during the maven / gradle build (e.g.
>> using the
>> : >>> Maven Shade plugin). Keep in mind that Lucene is also not suitable
>> for use
>>
>>
>> -Hoss
>> http://www.lucidworks.com/
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
Re: Lucene 9.0 Java module system support
Posted by David Ryan <da...@livemedia.com.au>.
Hi Uwe,
Thanks for taking the time out of your Easter break to respond. I've
created the following Jira tickets for preparatory tasks:
LUCENE-9317 <https://issues.apache.org/jira/browse/LUCENE-9317> Resolve
package name conflicts for StandardAnalyzer to allow Java module system
support
LUCENE-9318 <https://issues.apache.org/jira/browse/LUCENE-9318> Fix Codec
API to not rely on package-private classes as part of changes to support
java module system
LUCENE-9319 <https://issues.apache.org/jira/browse/LUCENE-9319> Clean up
Sandbox project by retiring/delete functionality or move it to Lucene core
I've also created the following for adding module-info.java:
LUCENE-9320 <https://issues.apache.org/jira/browse/LUCENE-9320> Enable
support the Java module system by adding module-info to projects
I've checked out the master branch and starting to look at setting up the
development environment using Gradle to see if I can assist with a few
items. I notice Gradle hasn't been set up for Eclipse yet (maybe something
I can help with). My first roadblock to getting setup is understanding the
cyclic dependencies between core, codecs and the test-framework. Given IRC
is not really used from what I can see, and the slack channel is private to
people with @apache.org emails. Is there any good place to get basic help
on getting started with the codebase without spamming this email list?
Thanks,
David.
On Sat, Apr 11, 2020 at 9:14 PM Uwe Schindler <uw...@thetaphi.de> wrote:
> Hi,
>
>
>
> I am currently on Easter vacation, but just wanted to give some feedback:
>
> They wy you are doing it works at the moment, because you don’t have
> anything to do with service providers. You don’t use custom codecs (outside
> lucene-core.jar) and you just ignored the META-IF/services folder. Your
> example works, as Lucene finds everything that is needed in the core.jar,
> so the “legacy” service loading works (lucene-core.jar find its own
> services in lucene-core.jar). As soon as you would add
> lucene-backwards-codecs.jar, it would break unfixable, unless you copy the
> whole backwards codecs into core. This is completely unfixable in Lucene
> 8.x, as the new service-loader interface requires Java 9+.
>
>
>
> But you may not yet have noticed that I already started to do preparatory
> work and migrated in master (Lucene 9.0) the service provider to use the
> Java Runtime classes and we have thrown away our own service loader (which
> cannot cross module boundaries). The fix was merged not long ago:
> https://issues.apache.org/jira/browse/LUCENE-9281 - this cannot be
> backported to Java 8 / Lucene 8, as the reason for the home-made
> SPIClassIterator was exactly the missing features (and some classpath
> ordering issues in 3rd party JVMs like IBM J9). Those issues are fixed in
> Java 11 (also allowing to instantiate the classes on your own). When you
> are testing your “hack patch” with Lucene’s master branch you may succeed
> with also loading classes from the backwards codecs – no guarantees yet!)
>
>
>
> About StandardAnalyzer: Unfortunately I aggressively complained a while
> back when Mike McCandless wanted to move standard analyzer out of the
> analysis package into core (“for convenience”). This was a bad step, and
> IMHO we should revert that or completely rename the packages and
> everything. The problem here is: As the analysis services are only part of
> lucene-analyzers, we had to leave the factory classes there, but move the
> implementation classes in core. The package has to be the same. The only
> way around that is to move the analysis factory framework also to core (I
> would not be against that). This would include all factory base classes and
> the service loading stuff. Then we can move standard analyzer and some of
> the filters/tokenizers including their factories to core an that problem
> would be solved.
>
>
>
> My plan to move to modules is the following:
>
>
>
> (1) Do preparatory work:
>
> - Retire SPIClassIterator (DONE).
> - Add some preparatory issues to cleanup class hierarchy: Move
> Analysis SPI to core / remove StandardAnalyzer and related classes out of
> core back to anaysis
> - Fix Codec API to no rely on package-private classes, so we can have
> a completely public API with abstract classes for codecs, so stuff in
> backwards-codecs does not need to have access to package private stuff in
> core.
> - Cleanup sandbox to prefix all classes there with “sandbox” package
> and where needed remove package-private access. If it’s needed for internal
> access, WTF: Just move the stuff to core! We have a new version 9.0, so
> either retire/delete Sandbox stuff or make it part of Lucene core.
>
> (2) Wait for the Gradle Build to be finalized, because including Module
> stuff into the current Ant build won’t work
>
> - Ant build must be retired!
>
> (3) Make Lucene real modules
>
> - As we are on Java 11 already, add module-info.java everywhere
> - Fix gradle build to create and test modules (Latest Gradle needed)
> - Migrate all META-INF/services/* to module-info.java (before doing
> this, figure out of the META-INF files must stays for non-module usage, or
> if Java is clever enough to also look into module descriptor to find
> services). We may need all services at both locations (for module or
> classpath usage; we need a build helper to check that it’s in-line)
>
>
>
> I don’t want Lucene work with automodules in Java 11, it should be fully
> modularized.
>
>
>
> If you want to help and express interest: Please open an issue for the
> preparatory work listing the cases of same-package in different jar files.
> I would split this up as described above: Analysis issues with
> standardanalyzer, codecs pkg-private apis, sandbox.
>
>
>
> Uwe
>
>
>
> -----
>
> Uwe Schindler
>
> Achterdiek 19, D-28357 Bremen
>
> https://www.thetaphi.de
>
> eMail: uwe@thetaphi.de
>
>
>
> *From:* David Ryan <da...@livemedia.com.au>
> *Sent:* Saturday, April 11, 2020 9:30 AM
> *To:* dev@lucene.apache.org
> *Subject:* Re: Lucene 9.0 Java module system support
>
>
>
>
>
> Thanks, I had probably missed some of what Uwe was saying in regards to
> the limitation of what would be possible even if the suggested changes were
> made. Given your points and the fact that it would take a while to have any
> of the changes filter into a release version, I decided to develop a
> patch-lucene.sh script to validate if the changes would allow me to access
> the required parts of Lucene. While it may not provide the full
> functionality provided by SPI including analysis chains and codecs, I
> worked out it will allow the use of the basics.
>
>
>
> The patch script below adds Automatic module names to the four Lucene
> libraries I needed (not strictly required, but I was interested to check if
> it would work). For now, any duplicate packages I identified have been
> moved into the core library (as identified in suggested changes). Given,
> I'm not using the SPI functionality, I simply deleted the standard
> analysers from the services list (required or the module system complains
> they are missing). Thankfully Gradle doesn't validate jars in the cache so
> the patch script only needs to be run once.
>
>
>
> Once the patch script was applied, I was able to use Ngram analysers and
> the query parser without issue. This will provide the essentials of what we
> needed for an embedded solution (location indexing, short text indexing
> with ngrams tokenizer and lower case filter, searching by location with
> nearest, bounding box and radius search, and partial text searches using
> the query parser). Of course, I've only scratched the surface of the APIs
> and right now it isn't clear if/when the SPI functionality might cause
> further issues or be required.
>
>
>
> So, I've validated that the suggested changes would allow basic
> functionality to work under the Java module system. If there's anything
> else I can do to help progress the suggested changes please let me know.
>
>
>
> Regards,
>
> David.
>
>
>
>
>
> --------- patch-lucene.sh -- works on osx ------
>
> mkdir -p patch-lucene/core
> mkdir -p patch-lucene/analyzers
> mkdir -p patch-lucene/sandbox
> mkdir -p patch-lucene/queryparser
> cd patch-lucene
>
> # copy the jars from the gradle cache.
> cp
> ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-core/8.5.0/3f9ea85fff4fc3f7c83869dddb9b0ef7818c0cae/lucene-core-8.5.0.jar
> .
> cp
> ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-analyzers-common/8.5.0/7156f2e545fd6e7faaee4781d15eb60cf5f07646/lucene-analyzers-common-8.5.0.jar
> .
> cp
> ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-sandbox/8.5.0/2b275921f2fd92b15b4f1a2a565467c3fa221ef9/lucene-sandbox-8.5.0.jar
> .
> cp
> ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queryparser/8.5.0/13c38f39b1a7d10c4749ba789fa95da5868d4885/lucene-queryparser-8.5.0.jar
> .
>
> # Add automatic module name to core.
> cd core
> jar -xf ../lucene-core-8.5.0.jar
> sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name:
> org.apache.lucene.core'$'\n' META-INF/MANIFEST.MF
>
> # Add automiatc module name, move standard analysis classes into core.
> Remove any classes in standard from service lists.
> cd ../analyzers
> jar -xf ../lucene-analyzers-common-8.5.0.jar
> sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name:
> org.apache.lucene.analyzers.common'$'\n' META-INF/MANIFEST.MF
> sed -i '' -e '/standard/d' META-INF/services/*
> mv org/apache/lucene/analysis/standard/*
> ../core/org/apache/lucene/analysis/standard
> rmdir org/apache/lucene/analysis/standard
> jar -cfm ../lucene-analyzers-common-8.5.0-fix.jar META-INF/MANIFEST.MF .
>
> # Add automatic module name, move search and document packages into core.
> move Java 9 specific search into core.
> cd ../sandbox
> jar -xf ../lucene-sandbox-8.5.0.jar
> sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name:
> org.apache.lucene.sandbox'$'\n' META-INF/MANIFEST.MF
> mv org/apache/lucene/search/* ../core/org/apache/lucene/search
> rmdir org/apache/lucene/search
> mv org/apache/lucene/document/* ../core/org/apache/lucene/document
> rmdir org/apache/lucene/document
> mv META-INF/versions/9/org/apache/lucene/search/*
> ../core/META-INF/versions/9/org/apache/lucene/search
> rm -rf META-INF/versions
> jar -cfm ../lucene-sandbox-8.5.0-fix.jar META-INF/MANIFEST.MF .
>
> # Package up core with changes.
> cd ../core
> jar -cfm ../lucene-core-8.5.0-fix.jar META-INF/MANIFEST.MF .
>
> # Add automatic module name.
> cd ../queryparser
> jar -xf ../lucene-queryparser-8.5.0.jar
> sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name:
> org.apache.lucene.queryparser'$'\n' META-INF/MANIFEST.MF
> jar -cfm ../lucene-queryparser-8.5.0-fix.jar META-INF/MANIFEST.MF .
>
> # Copy the fixed versions back into gradle cache.
> cd ..
> cp lucene-core-8.5.0-fix.jar
> ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-core/8.5.0/3f9ea85fff4fc3f7c83869dddb9b0ef7818c0cae/lucene-core-8.5.0.jar
> cp lucene-analyzers-common-8.5.0-fix.jar
> ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-analyzers-common/8.5.0/7156f2e545fd6e7faaee4781d15eb60cf5f07646/lucene-analyzers-common-8.5.0.jar
> cp lucene-sandbox-8.5.0-fix.jar
> ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-sandbox/8.5.0/2b275921f2fd92b15b4f1a2a565467c3fa221ef9/lucene-sandbox-8.5.0.jar
> cp lucene-queryparser-8.5.0-fix.jar
> ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queryparser/8.5.0/13c38f39b1a7d10c4749ba789fa95da5868d4885/lucene-queryparser-8.5.0.jar
>
> ------------
>
>
>
> On Sat, Apr 11, 2020 at 11:21 AM Chris Hostetter <ho...@fucit.org>
> wrote:
>
>
> : If the changes I proposed are still viewed as having too many downstream
> : impacts, my fallback position will be to patch the jars. This involves
> : using the gradle import system to get the jars from Maven, then using a
> : patch script to manually unzip the jars, move the offending classes into
> : other jars which share the same package name and rezip. So far, I've been
>
> I'm no expert here, but i trust that Uwe is, and i feel like your followup
> questions/suggests have still avoided his primary point about *why*
> Lucene/Solr hasn't attempted jigsaw modulariation...
>
> : >>> There is currently some preparatory things to move forward with
> modules,
> : >>> so although you might be able to actually compile Lucene with module
> system
> : >>> (by limiting to a subset of JAR files), it currently won’t work
> : >>> cross-module due to the way how it handles ServiceLoader interfaces
> : >>> (codecs, postings formats, analyzers, see
> : >>> https://issues.apache.org/jira/browse/LUCENE-9281). The only way to
> : >>> make it work at runtime is to put all of Lucene into one module.
>
> ...so, IIUC, even if you patch the *current* jars, any Lucene code you use
> that depends on SPI (like analysis chains, codecs, etc...) isn't going to
> work unless follow Uwe's primary suggestion for folks who care about
> modules...
>
> : >>> Th general recommendation is to combine all required Lucene libraries
> : >>> into a separate JAR file during the maven / gradle build (e.g. using
> the
> : >>> Maven Shade plugin). Keep in mind that Lucene is also not suitable
> for use
>
>
> -Hoss
> http://www.lucidworks.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
RE: Lucene 9.0 Java module system support
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
I am currently on Easter vacation, but just wanted to give some feedback:
They wy you are doing it works at the moment, because you don’t have anything to do with service providers. You don’t use custom codecs (outside lucene-core.jar) and you just ignored the META-IF/services folder. Your example works, as Lucene finds everything that is needed in the core.jar, so the “legacy” service loading works (lucene-core.jar find its own services in lucene-core.jar). As soon as you would add lucene-backwards-codecs.jar, it would break unfixable, unless you copy the whole backwards codecs into core. This is completely unfixable in Lucene 8.x, as the new service-loader interface requires Java 9+.
But you may not yet have noticed that I already started to do preparatory work and migrated in master (Lucene 9.0) the service provider to use the Java Runtime classes and we have thrown away our own service loader (which cannot cross module boundaries). The fix was merged not long ago: <https://issues.apache.org/jira/browse/LUCENE-9281> https://issues.apache.org/jira/browse/LUCENE-9281 - this cannot be backported to Java 8 / Lucene 8, as the reason for the home-made SPIClassIterator was exactly the missing features (and some classpath ordering issues in 3rd party JVMs like IBM J9). Those issues are fixed in Java 11 (also allowing to instantiate the classes on your own). When you are testing your “hack patch” with Lucene’s master branch you may succeed with also loading classes from the backwards codecs – no guarantees yet!)
About StandardAnalyzer: Unfortunately I aggressively complained a while back when Mike McCandless wanted to move standard analyzer out of the analysis package into core (“for convenience”). This was a bad step, and IMHO we should revert that or completely rename the packages and everything. The problem here is: As the analysis services are only part of lucene-analyzers, we had to leave the factory classes there, but move the implementation classes in core. The package has to be the same. The only way around that is to move the analysis factory framework also to core (I would not be against that). This would include all factory base classes and the service loading stuff. Then we can move standard analyzer and some of the filters/tokenizers including their factories to core an that problem would be solved.
My plan to move to modules is the following:
(1) Do preparatory work:
* Retire SPIClassIterator (DONE).
* Add some preparatory issues to cleanup class hierarchy: Move Analysis SPI to core / remove StandardAnalyzer and related classes out of core back to anaysis
* Fix Codec API to no rely on package-private classes, so we can have a completely public API with abstract classes for codecs, so stuff in backwards-codecs does not need to have access to package private stuff in core.
* Cleanup sandbox to prefix all classes there with “sandbox” package and where needed remove package-private access. If it’s needed for internal access, WTF: Just move the stuff to core! We have a new version 9.0, so either retire/delete Sandbox stuff or make it part of Lucene core.
(2) Wait for the Gradle Build to be finalized, because including Module stuff into the current Ant build won’t work
* Ant build must be retired!
(3) Make Lucene real modules
* As we are on Java 11 already, add module-info.java everywhere
* Fix gradle build to create and test modules (Latest Gradle needed)
* Migrate all META-INF/services/* to module-info.java (before doing this, figure out of the META-INF files must stays for non-module usage, or if Java is clever enough to also look into module descriptor to find services). We may need all services at both locations (for module or classpath usage; we need a build helper to check that it’s in-line)
I don’t want Lucene work with automodules in Java 11, it should be fully modularized.
If you want to help and express interest: Please open an issue for the preparatory work listing the cases of same-package in different jar files. I would split this up as described above: Analysis issues with standardanalyzer, codecs pkg-private apis, sandbox.
Uwe
-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: uwe@thetaphi.de
From: David Ryan <da...@livemedia.com.au>
Sent: Saturday, April 11, 2020 9:30 AM
To: dev@lucene.apache.org
Subject: Re: Lucene 9.0 Java module system support
Thanks, I had probably missed some of what Uwe was saying in regards to the limitation of what would be possible even if the suggested changes were made. Given your points and the fact that it would take a while to have any of the changes filter into a release version, I decided to develop a patch-lucene.sh script to validate if the changes would allow me to access the required parts of Lucene. While it may not provide the full functionality provided by SPI including analysis chains and codecs, I worked out it will allow the use of the basics.
The patch script below adds Automatic module names to the four Lucene libraries I needed (not strictly required, but I was interested to check if it would work). For now, any duplicate packages I identified have been moved into the core library (as identified in suggested changes). Given, I'm not using the SPI functionality, I simply deleted the standard analysers from the services list (required or the module system complains they are missing). Thankfully Gradle doesn't validate jars in the cache so the patch script only needs to be run once.
Once the patch script was applied, I was able to use Ngram analysers and the query parser without issue. This will provide the essentials of what we needed for an embedded solution (location indexing, short text indexing with ngrams tokenizer and lower case filter, searching by location with nearest, bounding box and radius search, and partial text searches using the query parser). Of course, I've only scratched the surface of the APIs and right now it isn't clear if/when the SPI functionality might cause further issues or be required.
So, I've validated that the suggested changes would allow basic functionality to work under the Java module system. If there's anything else I can do to help progress the suggested changes please let me know.
Regards,
David.
--------- patch-lucene.sh -- works on osx ------
mkdir -p patch-lucene/core
mkdir -p patch-lucene/analyzers
mkdir -p patch-lucene/sandbox
mkdir -p patch-lucene/queryparser
cd patch-lucene
# copy the jars from the gradle cache.
cp ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-core/8.5.0/3f9ea85fff4fc3f7c83869dddb9b0ef7818c0cae/lucene-core-8.5.0.jar .
cp ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-analyzers-common/8.5.0/7156f2e545fd6e7faaee4781d15eb60cf5f07646/lucene-analyzers-common-8.5.0.jar .
cp ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-sandbox/8.5.0/2b275921f2fd92b15b4f1a2a565467c3fa221ef9/lucene-sandbox-8.5.0.jar .
cp ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queryparser/8.5.0/13c38f39b1a7d10c4749ba789fa95da5868d4885/lucene-queryparser-8.5.0.jar .
# Add automatic module name to core.
cd core
jar -xf ../lucene-core-8.5.0.jar
sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name: org.apache.lucene.core'$'\n' META-INF/MANIFEST.MF
# Add automiatc module name, move standard analysis classes into core. Remove any classes in standard from service lists.
cd ../analyzers
jar -xf ../lucene-analyzers-common-8.5.0.jar
sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name: org.apache.lucene.analyzers.common'$'\n' META-INF/MANIFEST.MF
sed -i '' -e '/standard/d' META-INF/services/*
mv org/apache/lucene/analysis/standard/* ../core/org/apache/lucene/analysis/standard
rmdir org/apache/lucene/analysis/standard
jar -cfm ../lucene-analyzers-common-8.5.0-fix.jar META-INF/MANIFEST.MF .
# Add automatic module name, move search and document packages into core. move Java 9 specific search into core.
cd ../sandbox
jar -xf ../lucene-sandbox-8.5.0.jar
sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name: org.apache.lucene.sandbox'$'\n' META-INF/MANIFEST.MF
mv org/apache/lucene/search/* ../core/org/apache/lucene/search
rmdir org/apache/lucene/search
mv org/apache/lucene/document/* ../core/org/apache/lucene/document
rmdir org/apache/lucene/document
mv META-INF/versions/9/org/apache/lucene/search/* ../core/META-INF/versions/9/org/apache/lucene/search
rm -rf META-INF/versions
jar -cfm ../lucene-sandbox-8.5.0-fix.jar META-INF/MANIFEST.MF .
# Package up core with changes.
cd ../core
jar -cfm ../lucene-core-8.5.0-fix.jar META-INF/MANIFEST.MF .
# Add automatic module name.
cd ../queryparser
jar -xf ../lucene-queryparser-8.5.0.jar
sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name: org.apache.lucene.queryparser'$'\n' META-INF/MANIFEST.MF
jar -cfm ../lucene-queryparser-8.5.0-fix.jar META-INF/MANIFEST.MF .
# Copy the fixed versions back into gradle cache.
cd ..
cp lucene-core-8.5.0-fix.jar ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-core/8.5.0/3f9ea85fff4fc3f7c83869dddb9b0ef7818c0cae/lucene-core-8.5.0.jar
cp lucene-analyzers-common-8.5.0-fix.jar ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-analyzers-common/8.5.0/7156f2e545fd6e7faaee4781d15eb60cf5f07646/lucene-analyzers-common-8.5.0.jar
cp lucene-sandbox-8.5.0-fix.jar ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-sandbox/8.5.0/2b275921f2fd92b15b4f1a2a565467c3fa221ef9/lucene-sandbox-8.5.0.jar
cp lucene-queryparser-8.5.0-fix.jar ~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queryparser/8.5.0/13c38f39b1a7d10c4749ba789fa95da5868d4885/lucene-queryparser-8.5.0.jar
------------
On Sat, Apr 11, 2020 at 11:21 AM Chris Hostetter <hossman_lucene@fucit.org <ma...@fucit.org> > wrote:
: If the changes I proposed are still viewed as having too many downstream
: impacts, my fallback position will be to patch the jars. This involves
: using the gradle import system to get the jars from Maven, then using a
: patch script to manually unzip the jars, move the offending classes into
: other jars which share the same package name and rezip. So far, I've been
I'm no expert here, but i trust that Uwe is, and i feel like your followup
questions/suggests have still avoided his primary point about *why*
Lucene/Solr hasn't attempted jigsaw modulariation...
: >>> There is currently some preparatory things to move forward with modules,
: >>> so although you might be able to actually compile Lucene with module system
: >>> (by limiting to a subset of JAR files), it currently won’t work
: >>> cross-module due to the way how it handles ServiceLoader interfaces
: >>> (codecs, postings formats, analyzers, see
: >>> https://issues.apache.org/jira/browse/LUCENE-9281). The only way to
: >>> make it work at runtime is to put all of Lucene into one module.
...so, IIUC, even if you patch the *current* jars, any Lucene code you use
that depends on SPI (like analysis chains, codecs, etc...) isn't going to
work unless follow Uwe's primary suggestion for folks who care about
modules...
: >>> Th general recommendation is to combine all required Lucene libraries
: >>> into a separate JAR file during the maven / gradle build (e.g. using the
: >>> Maven Shade plugin). Keep in mind that Lucene is also not suitable for use
-Hoss
http://www.lucidworks.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org <ma...@lucene.apache.org>
For additional commands, e-mail: dev-help@lucene.apache.org <ma...@lucene.apache.org>
Re: Lucene 9.0 Java module system support
Posted by David Ryan <da...@livemedia.com.au>.
Thanks, I had probably missed some of what Uwe was saying in regards to the
limitation of what would be possible even if the suggested changes were
made. Given your points and the fact that it would take a while to have any
of the changes filter into a release version, I decided to develop a
patch-lucene.sh script to validate if the changes would allow me to access
the required parts of Lucene. While it may not provide the full
functionality provided by SPI including analysis chains and codecs, I
worked out it will allow the use of the basics.
The patch script below adds Automatic module names to the four Lucene
libraries I needed (not strictly required, but I was interested to check if
it would work). For now, any duplicate packages I identified have been
moved into the core library (as identified in suggested changes). Given,
I'm not using the SPI functionality, I simply deleted the standard
analysers from the services list (required or the module system complains
they are missing). Thankfully Gradle doesn't validate jars in the cache so
the patch script only needs to be run once.
Once the patch script was applied, I was able to use Ngram analysers and
the query parser without issue. This will provide the essentials of what we
needed for an embedded solution (location indexing, short text indexing
with ngrams tokenizer and lower case filter, searching by location with
nearest, bounding box and radius search, and partial text searches using
the query parser). Of course, I've only scratched the surface of the APIs
and right now it isn't clear if/when the SPI functionality might cause
further issues or be required.
So, I've validated that the suggested changes would allow basic
functionality to work under the Java module system. If there's anything
else I can do to help progress the suggested changes please let me know.
Regards,
David.
--------- patch-lucene.sh -- works on osx ------
mkdir -p patch-lucene/core
mkdir -p patch-lucene/analyzers
mkdir -p patch-lucene/sandbox
mkdir -p patch-lucene/queryparser
cd patch-lucene
# copy the jars from the gradle cache.
cp
~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-core/8.5.0/3f9ea85fff4fc3f7c83869dddb9b0ef7818c0cae/lucene-core-8.5.0.jar
.
cp
~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-analyzers-common/8.5.0/7156f2e545fd6e7faaee4781d15eb60cf5f07646/lucene-analyzers-common-8.5.0.jar
.
cp
~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-sandbox/8.5.0/2b275921f2fd92b15b4f1a2a565467c3fa221ef9/lucene-sandbox-8.5.0.jar
.
cp
~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queryparser/8.5.0/13c38f39b1a7d10c4749ba789fa95da5868d4885/lucene-queryparser-8.5.0.jar
.
# Add automatic module name to core.
cd core
jar -xf ../lucene-core-8.5.0.jar
sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name:
org.apache.lucene.core'$'\n' META-INF/MANIFEST.MF
# Add automiatc module name, move standard analysis classes into core.
Remove any classes in standard from service lists.
cd ../analyzers
jar -xf ../lucene-analyzers-common-8.5.0.jar
sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name:
org.apache.lucene.analyzers.common'$'\n' META-INF/MANIFEST.MF
sed -i '' -e '/standard/d' META-INF/services/*
mv org/apache/lucene/analysis/standard/*
../core/org/apache/lucene/analysis/standard
rmdir org/apache/lucene/analysis/standard
jar -cfm ../lucene-analyzers-common-8.5.0-fix.jar META-INF/MANIFEST.MF .
# Add automatic module name, move search and document packages into core.
move Java 9 specific search into core.
cd ../sandbox
jar -xf ../lucene-sandbox-8.5.0.jar
sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name:
org.apache.lucene.sandbox'$'\n' META-INF/MANIFEST.MF
mv org/apache/lucene/search/* ../core/org/apache/lucene/search
rmdir org/apache/lucene/search
mv org/apache/lucene/document/* ../core/org/apache/lucene/document
rmdir org/apache/lucene/document
mv META-INF/versions/9/org/apache/lucene/search/*
../core/META-INF/versions/9/org/apache/lucene/search
rm -rf META-INF/versions
jar -cfm ../lucene-sandbox-8.5.0-fix.jar META-INF/MANIFEST.MF .
# Package up core with changes.
cd ../core
jar -cfm ../lucene-core-8.5.0-fix.jar META-INF/MANIFEST.MF .
# Add automatic module name.
cd ../queryparser
jar -xf ../lucene-queryparser-8.5.0.jar
sed -i '' -e '/Created-By:/a\'$'\n''Automatic-Module-Name:
org.apache.lucene.queryparser'$'\n' META-INF/MANIFEST.MF
jar -cfm ../lucene-queryparser-8.5.0-fix.jar META-INF/MANIFEST.MF .
# Copy the fixed versions back into gradle cache.
cd ..
cp lucene-core-8.5.0-fix.jar
~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-core/8.5.0/3f9ea85fff4fc3f7c83869dddb9b0ef7818c0cae/lucene-core-8.5.0.jar
cp lucene-analyzers-common-8.5.0-fix.jar
~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-analyzers-common/8.5.0/7156f2e545fd6e7faaee4781d15eb60cf5f07646/lucene-analyzers-common-8.5.0.jar
cp lucene-sandbox-8.5.0-fix.jar
~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-sandbox/8.5.0/2b275921f2fd92b15b4f1a2a565467c3fa221ef9/lucene-sandbox-8.5.0.jar
cp lucene-queryparser-8.5.0-fix.jar
~/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queryparser/8.5.0/13c38f39b1a7d10c4749ba789fa95da5868d4885/lucene-queryparser-8.5.0.jar
------------
On Sat, Apr 11, 2020 at 11:21 AM Chris Hostetter <ho...@fucit.org>
wrote:
>
> : If the changes I proposed are still viewed as having too many downstream
> : impacts, my fallback position will be to patch the jars. This involves
> : using the gradle import system to get the jars from Maven, then using a
> : patch script to manually unzip the jars, move the offending classes into
> : other jars which share the same package name and rezip. So far, I've been
>
> I'm no expert here, but i trust that Uwe is, and i feel like your followup
> questions/suggests have still avoided his primary point about *why*
> Lucene/Solr hasn't attempted jigsaw modulariation...
>
> : >>> There is currently some preparatory things to move forward with
> modules,
> : >>> so although you might be able to actually compile Lucene with module
> system
> : >>> (by limiting to a subset of JAR files), it currently won’t work
> : >>> cross-module due to the way how it handles ServiceLoader interfaces
> : >>> (codecs, postings formats, analyzers, see
> : >>> https://issues.apache.org/jira/browse/LUCENE-9281). The only way to
> : >>> make it work at runtime is to put all of Lucene into one module.
>
> ...so, IIUC, even if you patch the *current* jars, any Lucene code you use
> that depends on SPI (like analysis chains, codecs, etc...) isn't going to
> work unless follow Uwe's primary suggestion for folks who care about
> modules...
>
> : >>> Th general recommendation is to combine all required Lucene libraries
> : >>> into a separate JAR file during the maven / gradle build (e.g. using
> the
> : >>> Maven Shade plugin). Keep in mind that Lucene is also not suitable
> for use
>
>
> -Hoss
> http://www.lucidworks.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
Re: Lucene 9.0 Java module system support
Posted by Chris Hostetter <ho...@fucit.org>.
: If the changes I proposed are still viewed as having too many downstream
: impacts, my fallback position will be to patch the jars. This involves
: using the gradle import system to get the jars from Maven, then using a
: patch script to manually unzip the jars, move the offending classes into
: other jars which share the same package name and rezip. So far, I've been
I'm no expert here, but i trust that Uwe is, and i feel like your followup
questions/suggests have still avoided his primary point about *why*
Lucene/Solr hasn't attempted jigsaw modulariation...
: >>> There is currently some preparatory things to move forward with modules,
: >>> so although you might be able to actually compile Lucene with module system
: >>> (by limiting to a subset of JAR files), it currently won’t work
: >>> cross-module due to the way how it handles ServiceLoader interfaces
: >>> (codecs, postings formats, analyzers, see
: >>> https://issues.apache.org/jira/browse/LUCENE-9281). The only way to
: >>> make it work at runtime is to put all of Lucene into one module.
...so, IIUC, even if you patch the *current* jars, any Lucene code you use
that depends on SPI (like analysis chains, codecs, etc...) isn't going to
work unless follow Uwe's primary suggestion for folks who care about
modules...
: >>> Th general recommendation is to combine all required Lucene libraries
: >>> into a separate JAR file during the maven / gradle build (e.g. using the
: >>> Maven Shade plugin). Keep in mind that Lucene is also not suitable for use
-Hoss
http://www.lucidworks.com/
Re: Lucene 9.0 Java module system support
Posted by David Ryan <da...@livemedia.com.au>.
Just to be clear, I'm not suggesting that you or the Solr project should
use the module system. I agree that it is not for every project, however,
more and more projects and libraries are moving slowly to using the module
system or making their libraries compatible. I'm not suggesting you add
module-info.java classes in each of your libraries. The proposed changes
I've highlighted are the minimum changes required to allow me and anyone
else that chooses to use the module system to include Lucene libraries
without issues.
In regards to your observations, you are right, there are plenty of issues
moving to use the module system. We made the choice of moving a large
project to use the module system while switching to Java 11 last year. We
viewed this as paying forward the technical debt we thought we would
encounter sometime in the future. Overall, I'm pleased we made the jump.
While we do need to have a more complex start script (--add-modules,
--add-opens, --add-exports, etc), that has been a set and forget exercise
in the shell script and joins the GC, memory config, JMX and other config
that was there previously. I like that the module-info forces us to think
about how each sub-library in our system interfaces with others. We have a
way to go in closing off implementation details, but this can be tackled
over time.
Maybe surprisingly, this is the first library I've wanted to use that
hasn't got a plan to at least be compatible (ie make minor changes to add
an Automatic-module-name and clean up shared package names). Other
libraries I use successfully with the module system include, google guice,
apache commons, jackson, jersey, jetty, dropwizard metrics, etc. I've also
been able to make limited use of Lucene for spatial indexing by importing
only lucene-core and lucene-spatial only. However, if I want to use
anything past that it requires importing lucene-analyzers-common and
lucene-queryparser (which requires lucene-sandbox) and that is where I have
package name conflicts.
If the changes I proposed are still viewed as having too many downstream
impacts, my fallback position will be to patch the jars. This involves
using the gradle import system to get the jars from Maven, then using a
patch script to manually unzip the jars, move the offending classes into
other jars which share the same package name and rezip. So far, I've been
required to patch two libraries (https://github.com/jnr/jffi/issues/73 and
https://github.com/datastax/native-protocol/issues/31 ), of course, I'd
rather avoid this where possible.
Regards,
David.
On Fri, Apr 10, 2020 at 11:58 PM Gus Heck <gu...@gmail.com> wrote:
> While the module system sounds nice in theory, my experience is
> unfortunately that it is quite difficult to use for any application with
> many existing dependencies, 90% or more of projects don't use it and
> therefore wind up in the default module. I've wasted many hours trying to
> get it to work in fat jar packaging situations (JesterJ uses my fork of
> one-jar, uno-jar) and that is nearly impossible. In order to use it one has
> to be willing to issue a very large number of --add-opens or --add-imports
> directives on the command line, which totally obliterates the nice java
> -jar myJar.jar command-line invocation, and creates a *requirement* for a
> startup script which then also has to maintain that list of directives
> (more cost than gain, but tractable for cases where there is a startup
> script, intractable for fat jars that want to be executable).
>
> There are allegedly manifest attributes, and you can find reference to
> add-exports attribute if you dig long enough to find the archives emails
> that were part of the JCP process... or more recently here:
> https://www.jrebel.com/blog/java-9-modules-cheat-sheet... but they are
> not documented here:
> https://docs.oracle.com/en/java/javase/11/docs/specs/jar/jar.html and
> while add-exports seems to work in the manifest, add-opens did not (at
> least when I tried to use it). I have yet to find either attribute in any
> official documentation despite hours of looking.
>
> My general inclination has become that modules of negative value for any
> project that uses jars that were not written by the project or a project
> they control, or has zero external dependencies, which is sad because
> that's what it was supposed to help fix. The primary reason for this is the
> inability to see into the default module if you declare a module, and the
> error mentioned at the top of this thread, which leaves one re-packaging
> 3rd party libraries (more cost than gain, possible license implications).
>
> I've generally concluded it's best to avoid modules unless it's forced on
> me for some reason. I've yet to feel a module helped me but many many times
> I found that it got in the way.
>
> YMMV,
> Gus
>
> On Fri, Apr 10, 2020 at 2:30 AM David Ryan <da...@livemedia.com.au> wrote:
>
>>
>> Hi Uwe,
>>
>> As I mentioned in the original post, the main issue I've come across with
>> being able to support Java 11 modules is that Lucene doesn't cleanly
>> separate the use of package names across jar libraries which results in
>> errors like:
>>
>> "The package org.apache.lucene.analysis.standard is accessible from more
>> than one module: lucene.analyzers.common, lucene.core"
>>
>> I ran some analysis to see how extensive the problem is using the code
>> below (pass in the directory of unzipped Lucene 8.5.0 as argument).
>> Actually, the problem is really not that big and could be fixed. If you
>> ignore duplicates in lucene-test-framework-8.5.0.jar,
>> lucene-backward-codecs-8.5.0.jar and lucene-misc-8.5.0, you are left with
>> the following issues:
>>
>> package org.apache.lucene.analysis.standard
>> - lucene-core-8.5.0.jar
>> - lucene-analyzers-common-8.5.0.jar
>>
>> Would it be viable to rename package org.apache.lucene.analysis.standard
>> in lucene-analyzers-common to org.apache.lucene.analysis.classic or move
>> the classes into the core library?
>>
>> package org.apache.lucene.search
>> package org.apache.lucene.document
>> - lucene-core-8.5.0.jar
>> - lucene-sandbox-8.5.0.jar
>>
>> Would it be possible to move the search and document package in
>> lucene-sandbox into a sandbox sub-package? e.g.
>> org.apache.lucene.sandbox.search
>>
>> package org.apache.lucene.collation
>> package org.apache.lucene.collation.tokenattributes
>> - lucene-analyzers-common-8.5.0.jar
>> - lucene-analyzers-icu-8.5.0.jar
>>
>> Would it be possible to move collation into a collation.icu sub-package?
>> e.g. org.apache.lucene.collation.icu
>>
>> As I'm not deeply familiar with the Lucene code base, I'm don't know what
>> the flow-on effects of these changes would be. However, I'd be happy to
>> raise the JIRA issue(s) and prepare the changes/pull request.
>>
>> I believe with these changes many of the important Lucene libraries could
>> be cleanly brought into an application using the module system. Another
>> nice to have would be to add an Automatic-Module-Name entry to the
>> MANIFEST.MF of each jar to stop automated module names.
>>
>> As you mentioned the fall back is to create a lucene-all-8.5.0.jar and
>> combine all classes into a single large file. Given this is not currently
>> published on Maven, if there's no interest in making the above changes,
>> would it be possible to change the build system to publish a lucene-all
>> package?
>>
>> Regards,
>> David.
>>
>> -------- Full output of duplicated packages --------
>>
>> package org.apache.lucene.analysis.standard
>> - lucene-core-8.5.0.jar
>> - lucene-analyzers-common-8.5.0.jar
>> - lucene-test-framework-8.5.0.jar
>>
>> package org.apache.lucene.codecs.blockterms
>> - lucene-codecs-8.5.0.jar
>> - lucene-test-framework-8.5.0.jar
>>
>> package org.apache.lucene.codecs.lucene80
>> - lucene-core-8.5.0.jar
>> - lucene-backward-codecs-8.5.0.jar
>>
>> package org.apache.lucene.search.similarities
>> - lucene-core-8.5.0.jar
>> - lucene-test-framework-8.5.0.jar
>>
>> package org.apache.lucene.codecs.lucene70
>> - lucene-core-8.5.0.jar
>> - lucene-backward-codecs-8.5.0.jar
>>
>> package org.apache.lucene.store
>> - lucene-misc-8.5.0.jar
>> - lucene-core-8.5.0.jar
>> - lucene-test-framework-8.5.0.jar
>>
>> package org.apache.lucene.codecs.bloom
>> - lucene-codecs-8.5.0.jar
>> - lucene-test-framework-8.5.0.jar
>>
>> package org.apache.lucene.codecs.lucene50
>> - lucene-core-8.5.0.jar
>> - lucene-backward-codecs-8.5.0.jar
>>
>> package org.apache.lucene.search.spans
>> - lucene-core-8.5.0.jar
>> - lucene-test-framework-8.5.0.jar
>>
>> package org.apache.lucene.index
>> - lucene-misc-8.5.0.jar
>> - lucene-core-8.5.0.jar
>> - lucene-test-framework-8.5.0.jar
>>
>> package org.apache.lucene.util.fst
>> - lucene-misc-8.5.0.jar
>> - lucene-core-8.5.0.jar
>> - lucene-test-framework-8.5.0.jar
>>
>> package org.apache.lucene.analysis
>> - lucene-core-8.5.0.jar
>> - lucene-test-framework-8.5.0.jar
>>
>> package org.apache.lucene.collation
>> - lucene-analyzers-common-8.5.0.jar
>> - lucene-analyzers-icu-8.5.0.jar
>>
>> package org.apache.lucene.codecs.uniformsplit.sharedterms
>> - lucene-codecs-8.5.0.jar
>> - lucene-test-framework-8.5.0.jar
>>
>> package org.apache.lucene.geo
>> - lucene-core-8.5.0.jar
>> - lucene-test-framework-8.5.0.jar
>>
>> package org.apache.lucene.search
>> - lucene-misc-8.5.0.jar
>> - lucene-core-8.5.0.jar
>> - lucene-test-framework-8.5.0.jar
>> - lucene-sandbox-8.5.0.jar
>>
>> package org.apache.lucene.collation.tokenattributes
>> - lucene-analyzers-common-8.5.0.jar
>> - lucene-analyzers-icu-8.5.0.jar
>>
>> package org.apache.lucene.document
>> - lucene-misc-8.5.0.jar
>> - lucene-core-8.5.0.jar
>> - lucene-sandbox-8.5.0.jar
>>
>> package org.apache.lucene.codecs.compressing
>> - lucene-core-8.5.0.jar
>> - lucene-test-framework-8.5.0.jar
>>
>> package org.apache.lucene.codecs.uniformsplit
>> - lucene-codecs-8.5.0.jar
>> - lucene-test-framework-8.5.0.jar
>>
>> package org.apache.lucene.util
>> - lucene-misc-8.5.0.jar
>> - lucene-core-8.5.0.jar
>> - lucene-test-framework-8.5.0.jar
>>
>> package org.apache.lucene.codecs
>> - lucene-core-8.5.0.jar
>> - lucene-test-framework-8.5.0.jar
>> - lucene-backward-codecs-8.5.0.jar
>>
>> package org.apache.lucene.util.automaton
>> - lucene-core-8.5.0.jar
>> - lucene-test-framework-8.5.0.jar
>>
>>
>> -------
>>
>> import java.io.File;
>> import java.io.FileInputStream;
>> import java.io.IOException;
>> import java.util.ArrayList;
>> import java.util.HashMap;
>> import java.util.List;
>> import java.util.Map;
>> import java.util.Map.Entry;
>> import java.util.zip.ZipEntry;
>> import java.util.zip.ZipInputStream;
>>
>> public class ReadJars {
>>
>> public static void main(final String[] arguments) throws IOException {
>> System.out.println("Find duplicate packages: <path to jar
>> files>");
>>
>> Map<String, List<String>> duplicates = new HashMap<>();
>> List<File> files = new ArrayList<File>();
>>
>> // Find all files in sub-directory.
>> addFiles(files, arguments[0]);
>>
>> // Find all package duplicates.
>> findDuplicatePackages(duplicates, files);
>>
>> // Print package with duplicate packages
>> for (Entry<String, List<String>> entry : duplicates.entrySet()) {
>> System.out.println("\npackage " + entry.getKey());
>> for (String dup : entry.getValue()) {
>> System.out.println(" - " + dup);
>> }
>> }
>>
>> }
>>
>> private static void findDuplicatePackages(Map<String, List<String>>
>> duplicates, List<File> files) throws IOException {
>> Map<String, String> packageToJarMap = new HashMap<>();
>> for (File f : files) {
>> if (f.getAbsolutePath().endsWith(".jar")) {
>>
>> String file =
>> f.getAbsolutePath().substring(f.getAbsolutePath().lastIndexOf('/') + 1);
>> ZipInputStream zip = new ZipInputStream(new
>> FileInputStream(f));
>>
>> for (ZipEntry entry = zip.getNextEntry(); entry != null;
>> entry = zip.getNextEntry()) {
>> if (!entry.isDirectory() &&
>> entry.getName().endsWith(".class")) {
>> String className = entry.getName();
>> String packageName = "";
>> if (className.lastIndexOf('/') >= 0) {
>> packageName = className.substring(0,
>> className.lastIndexOf('/')).replace('/', '.');
>> }
>> className = className.replace('/', '.');
>>
>> // Only interested in Lucene packages.
>> if (className.startsWith("org.apache.lucene")) {
>>
>> String jar = packageToJarMap.get(packageName);
>> if (jar != null && !jar.equals(file)) {
>> List<String> duplicate =
>> duplicates.get(packageName);
>> if (duplicate == null) {
>> duplicate = new ArrayList<>();
>> duplicate.add(jar);
>> duplicate.add(file);
>> duplicates.put(packageName,
>> duplicate);
>> } else {
>> boolean found = false;
>> for (String dup : duplicate) {
>> if (file.equals(dup)) {
>> found = true;
>> }
>> }
>> if (!found) {
>> duplicate.add(file);
>> }
>> }
>> } else {
>> packageToJarMap.put(packageName, file);
>> }
>> }
>> }
>> }
>> zip.close();
>> }
>> }
>> }
>>
>> public static void addFiles(List<File> files, String directoryName) {
>> File directory = new File(directoryName);
>> for (File file : directory.listFiles()) {
>> if (file.isFile()) {
>> files.add(file);
>> } else if (file.isDirectory()) {
>> addFiles(files, file.getAbsolutePath());
>> }
>> }
>> }
>> }
>>
>> --------
>>
>>
>> On Tue, Mar 24, 2020 at 9:26 PM Uwe Schindler <uw...@thetaphi.de> wrote:
>>
>>> Hi,
>>>
>>>
>>>
>>> this is a known problem since many year and there are no plans to change
>>> this yet. The main reason for Java 11 support is not to introduce the
>>> module system, but instead use the new features of Java 11 and to get rid
>>> MR-JAR complications to make use of new intrinsics.
>>>
>>>
>>>
>>> Servers like Solr or Elasticsearch are shipped as an application, so the
>>> module system does not bring any benefit.
>>>
>>>
>>>
>>> Th general recommendation is to combine all required Lucene libraries
>>> into a separate JAR file during the maven / gradle build (e.g. using the
>>> Maven Shade plugin). Keep in mind that Lucene is also not suitable for use
>>> in other module systems like
>>>
>>>
>>>
>>> There is currently some preparatory things to move forward with modules,
>>> so although you might be able to actually compile Lucene with module system
>>> (by limiting to a subset of JAR files), it currently won’t work
>>> cross-module due to the way how it handles ServiceLoader interfaces
>>> (codecs, postings formats, analyzers, see
>>> https://issues.apache.org/jira/browse/LUCENE-9281). The only way to
>>> make it work at runtime is to put all of Lucene into one module.
>>>
>>>
>>>
>>> Uwe
>>>
>>>
>>>
>>> -----
>>>
>>> Uwe Schindler
>>>
>>> Achterdiek 19, D-28357 Bremen
>>>
>>> https://www.thetaphi.de
>>>
>>> eMail: uwe@thetaphi.de
>>>
>>>
>>>
>>> *From:* David Ryan <da...@livemedia.com.au>
>>> *Sent:* Tuesday, March 24, 2020 8:05 AM
>>> *To:* dev@lucene.apache.org
>>> *Subject:* Lucene 9.0 Java module system support
>>>
>>>
>>>
>>>
>>>
>>> Hi all,
>>>
>>>
>>>
>>> I've been investigating the use of Lucene as part of an application that
>>> uses the Java Module System. Initially, I used Gradle to bring in
>>> lucene-core, lucene-spatial and lucene-queries using version 8.5.0. This
>>> works with the automated module naming. However, bringing in
>>> lucene-queryparser (which depends on lucene-sandbox) or
>>> lucene-analyzers-common causes errors such as:
>>>
>>>
>>>
>>> "The package org.apache.lucene.analysis.standard is accessible from more
>>> than one module: lucene.analyzers.common, lucene.core"
>>>
>>>
>>>
>>> The Java module system does not handle different jar files using the
>>> same package name which occurs throughout multiple maven artifacts.
>>>
>>>
>>>
>>> Looking at the LUCENE issues, I found the task of moving to a minimum
>>> version of Java 11, however, this does not mention the ability to be
>>> compatible with the module system. I also checked the git repository and
>>> couldn't find the required changes to support the module system.
>>>
>>>
>>>
>>> https://issues.apache.org/jira/browse/LUCENE-8738
>>>
>>>
>>>
>>> I looked through the dev list recent history but could not find anything
>>> related. Are there any plans to support modules? Given, I saw there are a
>>> number of other breaking changes happening with the move to Lucene 9.0,
>>> would it be good to make those changes?
>>>
>>>
>>>
>>> Thanks,
>>>
>>> David.
>>>
>>>
>>>
>>
>
> --
> http://www.needhamsoftware.com (work)
> http://www.the111shift.com (play)
>
Re: Lucene 9.0 Java module system support
Posted by Gus Heck <gu...@gmail.com>.
While the module system sounds nice in theory, my experience is
unfortunately that it is quite difficult to use for any application with
many existing dependencies, 90% or more of projects don't use it and
therefore wind up in the default module. I've wasted many hours trying to
get it to work in fat jar packaging situations (JesterJ uses my fork of
one-jar, uno-jar) and that is nearly impossible. In order to use it one has
to be willing to issue a very large number of --add-opens or --add-imports
directives on the command line, which totally obliterates the nice java
-jar myJar.jar command-line invocation, and creates a *requirement* for a
startup script which then also has to maintain that list of directives
(more cost than gain, but tractable for cases where there is a startup
script, intractable for fat jars that want to be executable).
There are allegedly manifest attributes, and you can find reference to
add-exports attribute if you dig long enough to find the archives emails
that were part of the JCP process... or more recently here:
https://www.jrebel.com/blog/java-9-modules-cheat-sheet... but they are not
documented here:
https://docs.oracle.com/en/java/javase/11/docs/specs/jar/jar.html and while
add-exports seems to work in the manifest, add-opens did not (at least when
I tried to use it). I have yet to find either attribute in any official
documentation despite hours of looking.
My general inclination has become that modules of negative value for any
project that uses jars that were not written by the project or a project
they control, or has zero external dependencies, which is sad because
that's what it was supposed to help fix. The primary reason for this is the
inability to see into the default module if you declare a module, and the
error mentioned at the top of this thread, which leaves one re-packaging
3rd party libraries (more cost than gain, possible license implications).
I've generally concluded it's best to avoid modules unless it's forced on
me for some reason. I've yet to feel a module helped me but many many times
I found that it got in the way.
YMMV,
Gus
On Fri, Apr 10, 2020 at 2:30 AM David Ryan <da...@livemedia.com.au> wrote:
>
> Hi Uwe,
>
> As I mentioned in the original post, the main issue I've come across with
> being able to support Java 11 modules is that Lucene doesn't cleanly
> separate the use of package names across jar libraries which results in
> errors like:
>
> "The package org.apache.lucene.analysis.standard is accessible from more
> than one module: lucene.analyzers.common, lucene.core"
>
> I ran some analysis to see how extensive the problem is using the code
> below (pass in the directory of unzipped Lucene 8.5.0 as argument).
> Actually, the problem is really not that big and could be fixed. If you
> ignore duplicates in lucene-test-framework-8.5.0.jar,
> lucene-backward-codecs-8.5.0.jar and lucene-misc-8.5.0, you are left with
> the following issues:
>
> package org.apache.lucene.analysis.standard
> - lucene-core-8.5.0.jar
> - lucene-analyzers-common-8.5.0.jar
>
> Would it be viable to rename package org.apache.lucene.analysis.standard
> in lucene-analyzers-common to org.apache.lucene.analysis.classic or move
> the classes into the core library?
>
> package org.apache.lucene.search
> package org.apache.lucene.document
> - lucene-core-8.5.0.jar
> - lucene-sandbox-8.5.0.jar
>
> Would it be possible to move the search and document package in
> lucene-sandbox into a sandbox sub-package? e.g.
> org.apache.lucene.sandbox.search
>
> package org.apache.lucene.collation
> package org.apache.lucene.collation.tokenattributes
> - lucene-analyzers-common-8.5.0.jar
> - lucene-analyzers-icu-8.5.0.jar
>
> Would it be possible to move collation into a collation.icu sub-package?
> e.g. org.apache.lucene.collation.icu
>
> As I'm not deeply familiar with the Lucene code base, I'm don't know what
> the flow-on effects of these changes would be. However, I'd be happy to
> raise the JIRA issue(s) and prepare the changes/pull request.
>
> I believe with these changes many of the important Lucene libraries could
> be cleanly brought into an application using the module system. Another
> nice to have would be to add an Automatic-Module-Name entry to the
> MANIFEST.MF of each jar to stop automated module names.
>
> As you mentioned the fall back is to create a lucene-all-8.5.0.jar and
> combine all classes into a single large file. Given this is not currently
> published on Maven, if there's no interest in making the above changes,
> would it be possible to change the build system to publish a lucene-all
> package?
>
> Regards,
> David.
>
> -------- Full output of duplicated packages --------
>
> package org.apache.lucene.analysis.standard
> - lucene-core-8.5.0.jar
> - lucene-analyzers-common-8.5.0.jar
> - lucene-test-framework-8.5.0.jar
>
> package org.apache.lucene.codecs.blockterms
> - lucene-codecs-8.5.0.jar
> - lucene-test-framework-8.5.0.jar
>
> package org.apache.lucene.codecs.lucene80
> - lucene-core-8.5.0.jar
> - lucene-backward-codecs-8.5.0.jar
>
> package org.apache.lucene.search.similarities
> - lucene-core-8.5.0.jar
> - lucene-test-framework-8.5.0.jar
>
> package org.apache.lucene.codecs.lucene70
> - lucene-core-8.5.0.jar
> - lucene-backward-codecs-8.5.0.jar
>
> package org.apache.lucene.store
> - lucene-misc-8.5.0.jar
> - lucene-core-8.5.0.jar
> - lucene-test-framework-8.5.0.jar
>
> package org.apache.lucene.codecs.bloom
> - lucene-codecs-8.5.0.jar
> - lucene-test-framework-8.5.0.jar
>
> package org.apache.lucene.codecs.lucene50
> - lucene-core-8.5.0.jar
> - lucene-backward-codecs-8.5.0.jar
>
> package org.apache.lucene.search.spans
> - lucene-core-8.5.0.jar
> - lucene-test-framework-8.5.0.jar
>
> package org.apache.lucene.index
> - lucene-misc-8.5.0.jar
> - lucene-core-8.5.0.jar
> - lucene-test-framework-8.5.0.jar
>
> package org.apache.lucene.util.fst
> - lucene-misc-8.5.0.jar
> - lucene-core-8.5.0.jar
> - lucene-test-framework-8.5.0.jar
>
> package org.apache.lucene.analysis
> - lucene-core-8.5.0.jar
> - lucene-test-framework-8.5.0.jar
>
> package org.apache.lucene.collation
> - lucene-analyzers-common-8.5.0.jar
> - lucene-analyzers-icu-8.5.0.jar
>
> package org.apache.lucene.codecs.uniformsplit.sharedterms
> - lucene-codecs-8.5.0.jar
> - lucene-test-framework-8.5.0.jar
>
> package org.apache.lucene.geo
> - lucene-core-8.5.0.jar
> - lucene-test-framework-8.5.0.jar
>
> package org.apache.lucene.search
> - lucene-misc-8.5.0.jar
> - lucene-core-8.5.0.jar
> - lucene-test-framework-8.5.0.jar
> - lucene-sandbox-8.5.0.jar
>
> package org.apache.lucene.collation.tokenattributes
> - lucene-analyzers-common-8.5.0.jar
> - lucene-analyzers-icu-8.5.0.jar
>
> package org.apache.lucene.document
> - lucene-misc-8.5.0.jar
> - lucene-core-8.5.0.jar
> - lucene-sandbox-8.5.0.jar
>
> package org.apache.lucene.codecs.compressing
> - lucene-core-8.5.0.jar
> - lucene-test-framework-8.5.0.jar
>
> package org.apache.lucene.codecs.uniformsplit
> - lucene-codecs-8.5.0.jar
> - lucene-test-framework-8.5.0.jar
>
> package org.apache.lucene.util
> - lucene-misc-8.5.0.jar
> - lucene-core-8.5.0.jar
> - lucene-test-framework-8.5.0.jar
>
> package org.apache.lucene.codecs
> - lucene-core-8.5.0.jar
> - lucene-test-framework-8.5.0.jar
> - lucene-backward-codecs-8.5.0.jar
>
> package org.apache.lucene.util.automaton
> - lucene-core-8.5.0.jar
> - lucene-test-framework-8.5.0.jar
>
>
> -------
>
> import java.io.File;
> import java.io.FileInputStream;
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.HashMap;
> import java.util.List;
> import java.util.Map;
> import java.util.Map.Entry;
> import java.util.zip.ZipEntry;
> import java.util.zip.ZipInputStream;
>
> public class ReadJars {
>
> public static void main(final String[] arguments) throws IOException {
> System.out.println("Find duplicate packages: <path to jar files>");
>
> Map<String, List<String>> duplicates = new HashMap<>();
> List<File> files = new ArrayList<File>();
>
> // Find all files in sub-directory.
> addFiles(files, arguments[0]);
>
> // Find all package duplicates.
> findDuplicatePackages(duplicates, files);
>
> // Print package with duplicate packages
> for (Entry<String, List<String>> entry : duplicates.entrySet()) {
> System.out.println("\npackage " + entry.getKey());
> for (String dup : entry.getValue()) {
> System.out.println(" - " + dup);
> }
> }
>
> }
>
> private static void findDuplicatePackages(Map<String, List<String>>
> duplicates, List<File> files) throws IOException {
> Map<String, String> packageToJarMap = new HashMap<>();
> for (File f : files) {
> if (f.getAbsolutePath().endsWith(".jar")) {
>
> String file =
> f.getAbsolutePath().substring(f.getAbsolutePath().lastIndexOf('/') + 1);
> ZipInputStream zip = new ZipInputStream(new
> FileInputStream(f));
>
> for (ZipEntry entry = zip.getNextEntry(); entry != null;
> entry = zip.getNextEntry()) {
> if (!entry.isDirectory() &&
> entry.getName().endsWith(".class")) {
> String className = entry.getName();
> String packageName = "";
> if (className.lastIndexOf('/') >= 0) {
> packageName = className.substring(0,
> className.lastIndexOf('/')).replace('/', '.');
> }
> className = className.replace('/', '.');
>
> // Only interested in Lucene packages.
> if (className.startsWith("org.apache.lucene")) {
>
> String jar = packageToJarMap.get(packageName);
> if (jar != null && !jar.equals(file)) {
> List<String> duplicate =
> duplicates.get(packageName);
> if (duplicate == null) {
> duplicate = new ArrayList<>();
> duplicate.add(jar);
> duplicate.add(file);
> duplicates.put(packageName, duplicate);
> } else {
> boolean found = false;
> for (String dup : duplicate) {
> if (file.equals(dup)) {
> found = true;
> }
> }
> if (!found) {
> duplicate.add(file);
> }
> }
> } else {
> packageToJarMap.put(packageName, file);
> }
> }
> }
> }
> zip.close();
> }
> }
> }
>
> public static void addFiles(List<File> files, String directoryName) {
> File directory = new File(directoryName);
> for (File file : directory.listFiles()) {
> if (file.isFile()) {
> files.add(file);
> } else if (file.isDirectory()) {
> addFiles(files, file.getAbsolutePath());
> }
> }
> }
> }
>
> --------
>
>
> On Tue, Mar 24, 2020 at 9:26 PM Uwe Schindler <uw...@thetaphi.de> wrote:
>
>> Hi,
>>
>>
>>
>> this is a known problem since many year and there are no plans to change
>> this yet. The main reason for Java 11 support is not to introduce the
>> module system, but instead use the new features of Java 11 and to get rid
>> MR-JAR complications to make use of new intrinsics.
>>
>>
>>
>> Servers like Solr or Elasticsearch are shipped as an application, so the
>> module system does not bring any benefit.
>>
>>
>>
>> Th general recommendation is to combine all required Lucene libraries
>> into a separate JAR file during the maven / gradle build (e.g. using the
>> Maven Shade plugin). Keep in mind that Lucene is also not suitable for use
>> in other module systems like
>>
>>
>>
>> There is currently some preparatory things to move forward with modules,
>> so although you might be able to actually compile Lucene with module system
>> (by limiting to a subset of JAR files), it currently won’t work
>> cross-module due to the way how it handles ServiceLoader interfaces
>> (codecs, postings formats, analyzers, see
>> https://issues.apache.org/jira/browse/LUCENE-9281). The only way to make
>> it work at runtime is to put all of Lucene into one module.
>>
>>
>>
>> Uwe
>>
>>
>>
>> -----
>>
>> Uwe Schindler
>>
>> Achterdiek 19, D-28357 Bremen
>>
>> https://www.thetaphi.de
>>
>> eMail: uwe@thetaphi.de
>>
>>
>>
>> *From:* David Ryan <da...@livemedia.com.au>
>> *Sent:* Tuesday, March 24, 2020 8:05 AM
>> *To:* dev@lucene.apache.org
>> *Subject:* Lucene 9.0 Java module system support
>>
>>
>>
>>
>>
>> Hi all,
>>
>>
>>
>> I've been investigating the use of Lucene as part of an application that
>> uses the Java Module System. Initially, I used Gradle to bring in
>> lucene-core, lucene-spatial and lucene-queries using version 8.5.0. This
>> works with the automated module naming. However, bringing in
>> lucene-queryparser (which depends on lucene-sandbox) or
>> lucene-analyzers-common causes errors such as:
>>
>>
>>
>> "The package org.apache.lucene.analysis.standard is accessible from more
>> than one module: lucene.analyzers.common, lucene.core"
>>
>>
>>
>> The Java module system does not handle different jar files using the same
>> package name which occurs throughout multiple maven artifacts.
>>
>>
>>
>> Looking at the LUCENE issues, I found the task of moving to a minimum
>> version of Java 11, however, this does not mention the ability to be
>> compatible with the module system. I also checked the git repository and
>> couldn't find the required changes to support the module system.
>>
>>
>>
>> https://issues.apache.org/jira/browse/LUCENE-8738
>>
>>
>>
>> I looked through the dev list recent history but could not find anything
>> related. Are there any plans to support modules? Given, I saw there are a
>> number of other breaking changes happening with the move to Lucene 9.0,
>> would it be good to make those changes?
>>
>>
>>
>> Thanks,
>>
>> David.
>>
>>
>>
>
--
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)