You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "uschindler (via GitHub)" <gi...@apache.org> on 2023/03/06 12:11:58 UTC
[GitHub] [lucene] uschindler opened a new pull request, #12188: Alternative version: Implement MMapDirectory with Java 19/20 Project Panama Preview API
uschindler opened a new pull request, #12188:
URL: https://github.com/apache/lucene/pull/12188
This is an altenative version to PR #12042. The problem with the previous PR (and also the Java 19 version) is mainly the following issues:
- We need to add a toolkit entry for each Java version we want to support. As Panama-Foreign is not yet finished, any developer who wants to build Lucene needs to either download Java 19+20+21 to actually compile the JARs or the autodownloader of Gradle will do it.
- We need to patch the class files to remove the preview bits after compilation
The approach here is a new idea that came to me when I studied, how the `--release` flag is implemented in javac. It works like that: Java has basically a copy of all class files of previous Java versions in some specially designed JAR file. When you request `--release 11` from Java compiler, it will overlay the compile classpath wit this JAR file that contains all classes (only signatures). Of course this will spend a lot of space, so there are 2 optimizations: it removes all bytecode from those JAR files, so all class files in it have empty method bodies. In addition, it will only add modified class files. After each relaese of Java they build this special signatures JAR file and store it in the JDK folder. FYI, preview API are excluded there, this is why we can't compile.
The PR here implements the following:
- It stores a similar (but simplified) version of the relevant classes in a "panama-foreign-jdk19.apijar" and "panama-foreign-jdk20.apijar"
- The apijar files are regenerated by the "regenerate task" or by calling `gradlew :lucene:core:generatePanamaForeignApiJar19` and/or `gradlew :lucene:core:generatePanamaForeignApiJar20`. This task uses Gradle's toolkit API to launch the correct Java version. So only people who regenerate files need to have Java 19, 20, 21,... installed. For Java 20 at moment you need it really installed locally, as it is impossible to autodownload it.
- When generating the APIJAR (a zip file in standard JAR format) it uses ASM to load the class files from the current JVM using the small tool `ExtractForeignAPI.java` which is executed using the toolkit API. It will then strip all bytecode and also strips the `@PreviewFeature` annotation (this allows to compile against the class file without Java adding the preview flags to compilation result. The outputted class files also get the version number for Java 11 class files. It also removes the option for nesthosts (not needed).
- When we compile out Java 19, Java 20, Java 21 MR-JAR, it will compile our sourceSet with the normal Java 11 or Java 17 compiler, but supplies `--patch-module java.base=panama-foreign-jdk20.apijar`. It also open the new foreign package, so it gets visible.
This PR allows anybody to compile the MR-JAR classes without the JDK installed. We can also merge this PR now, as the JDK 20 release is already in RC phase, so the API is finalized. The APIJAR files don't get outdated anymore. We can also release Lucene now with JDK 20 support!
Downsides:
- We have 2 or 3 binary files in out Git checkout (they are each approx 20 Kilobytes). An alternative would be to dump a txt-only signatures file, but that complicates it really. We can do that for licensing purposes.
- Licensing is hard, this is the main reason why I asked for comments: Is it ok to have a JAR file with method signatures in our source checkout? There's no code included (bytecode stripped)! The license is the GNU GPL with Classpath extension, but we compile against that already-
If anybody has an idea, speak up. If we agree, this would be the easiest to support MR-JARs of MMapDirectory in future (also with Java 21 where it is not yet fully sure that the API gets to its final stage! ....unfortunately.... damn!
The same approach could be used for vector APIs in future. Please add a comment about if you agree to do it like that!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #12188: Alternative version: Implement MMapDirectory with Java 19/20 Project Panama Preview API
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #12188:
URL: https://github.com/apache/lucene/pull/12188#issuecomment-1457731545
Hi FYI, here is the possible way to handle those licensing issues. A local patch that I have at hand may add a `panama-foreign-jdk20.txt` file next to the `panama-foreigen-jdk20.apijar` with the following content, so it is there in plain text waht the JAR file contains:
```
// These API signatures of Panama Foreign API were extracted from JDK 20
// Do not modify this file or the corresponding .apijar file which is used for MR-JAR compilation!
// class version 55.0 (55)
// access flags 0x601
public abstract interface java/lang/foreign/Arena implements java/lang/foreign/SegmentAllocator java/lang/AutoCloseable {
// access flags 0x1
public default allocate(JJ)Ljava/lang/foreign/MemorySegment;
// access flags 0x401
public abstract scope()Ljava/lang/foreign/SegmentScope;
// access flags 0x401
public abstract close()V
// access flags 0x401
public abstract isCloseableBy(Ljava/lang/Thread;)Z
// access flags 0x9
public static openConfined()Ljava/lang/foreign/Arena;
// access flags 0x9
public static openShared()Ljava/lang/foreign/Arena;
}
// class version 55.0 (55)
// access flags 0x601
public abstract interface java/lang/foreign/FunctionDescriptor {
// access flags 0x401
// signature ()Ljava/util/Optional<Ljava/lang/foreign/MemoryLayout;>;
// declaration: java.util.Optional<java.lang.foreign.MemoryLayout> returnLayout()
public abstract returnLayout()Ljava/util/Optional;
[...]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on a diff in pull request #12188: Alternative version: Implement MMapDirectory with Java 19/20 Project Panama Preview API
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on code in PR #12188:
URL: https://github.com/apache/lucene/pull/12188#discussion_r1126384213
##########
gradle/generation/panama-foreign/ExtractForeignAPI.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import java.io.IOException;
+import java.net.URI;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.nio.file.attribute.FileTime;
+import java.time.Instant;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.zip.ZipEntry;
+import java.util.zip.ZipOutputStream;
+
+import org.objectweb.asm.AnnotationVisitor;
+import org.objectweb.asm.ClassReader;
+import org.objectweb.asm.ClassVisitor;
+import org.objectweb.asm.ClassWriter;
+import org.objectweb.asm.FieldVisitor;
+import org.objectweb.asm.MethodVisitor;
+import org.objectweb.asm.Opcodes;
+import org.objectweb.asm.Type;
+
+public final class ExtractForeignAPI {
+
+ private static final FileTime FIXED_FILEDATE = FileTime.from(Instant.parse("2022-01-01T00:00:00Z"));
Review Comment:
I added this to make the JAR files not differ based on actual date/time. We could use any other date (e.g., the release date of Java version). But thats unimportant so I used this one.
In addition below, the files in JAR are sorted to make it reproducible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] dweiss commented on a diff in pull request #12188: Alternative version: Implement MMapDirectory with Java 19/20 Project Panama Preview API
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on code in PR #12188:
URL: https://github.com/apache/lucene/pull/12188#discussion_r1127465746
##########
gradle/generation/panama-foreign.gradle:
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+def resources = scriptResources(buildscript)
+
+configure(project(":lucene:core")) {
+ ext {
+ apijars = file('src/generated/jdk');
+ panamaJavaVersions = [ 19, 20 ]
+ }
+
+ configurations {
+ apiextractor
+ }
+
+ dependencies {
+ apiextractor "org.ow2.asm:asm:${scriptDepVersions['asm']}"
+ }
+
+ for (jdkVersion : panamaJavaVersions) {
+ def task = tasks.create(name: "generatePanamaForeignApiJar${jdkVersion}", type: JavaExec) {
+ description "Regenerate the API-only JAR file with public Panama Foreign API from JDK ${jdkVersion}"
+ group "generation"
+
+ javaLauncher = javaToolchains.launcherFor {
+ languageVersion = JavaLanguageVersion.of(jdkVersion)
+ }
+
+ onlyIf {
+ try {
+ javaLauncher.get()
+ return true
+ } catch (Exception e) {
+ logger.warn('Launcher for Java {} is not available; skipping regeneration of Panama Foreign API JAR.', jdkVersion)
+ logger.warn('Error: {}', e.cause?.message)
+ logger.warn("Please make sure to point env 'JAVA{}_HOME' to exactly JDK version {} or enable Gradle toolchain auto-download.", jdkVersion, jdkVersion)
+ return false
Review Comment:
Ok, clear. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #12188: Alternative version: Implement MMapDirectory with Java 19/20 Project Panama Preview API
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #12188:
URL: https://github.com/apache/lucene/pull/12188#issuecomment-1456053196
> The same approach could be used for vector APIs in future. Please add a comment about if you agree to do it like that!
when the vector api finally comes out, don't interrupt my shuffleboard match to tell me about it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #12188: Alternative version: Implement MMapDirectory with Java 19/20 Project Panama Preview API
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #12188:
URL: https://github.com/apache/lucene/pull/12188#issuecomment-1456075160
As you see this PR has passed all Github precommit CI checks (it does not need any Java version installed except the compilation one, not even 20).
As you see from the commit I started this approach already around new year's eve. I regenerated the APIJAR files today with the RC candidate. No changes (I only had to fix a bug later regarding the allowedSubclasses attribute that is now stripped).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #12188: Alternative version: Implement MMapDirectory with Java 19/20 Project Panama Preview API
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #12188:
URL: https://github.com/apache/lucene/pull/12188#issuecomment-1457689158
> I like it, Uwe. I have no idea about legal implications (whether interfaces alone are something that can be copied over).
>
> The only thing I've been wondering about is this: if these apis are still shaping and something changes upstream in the JDK, this change would go unnoticed. Do we rely on a periodic refresh of those API jars only? Or should we actually verify (in check) that the APIs are identical if the given JDK is available?
For JDK 19 and JDK 20 the "API changes" phase is over. The final release candidate is out and if those would be rebuilt or get a new release, the API won't change anymore. Of course we can execute "regenerate" for safety on March 21, but the above files were generated with the version to be released on that day already.
I would only add Java 21 once we are at a place when the API is finalized, which is not yet the sagte.
Chronological checks are not needed, as once JDK 19 / 20 are/were released the APIs are final.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #12188: Alternative version: Implement MMapDirectory with Java 19/20 Project Panama Preview API
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #12188:
URL: https://github.com/apache/lucene/pull/12188#issuecomment-1457813085
I further minimized file sizes by removing contents of hidden classes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #12188: Alternative version: Implement MMapDirectory with Java 19/20 Project Panama Preview API
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #12188:
URL: https://github.com/apache/lucene/pull/12188#issuecomment-1458208528
Hi @dweiss,
I added a README.md: https://github.com/apache/lucene/blob/9cad5756461bf35cd56ce0d88a434884c89a16e8/lucene/core/src/generated/jdk/README.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #12188: Alternative version: Implement MMapDirectory with Java 19/20 Project Panama Preview API
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #12188:
URL: https://github.com/apache/lucene/pull/12188#issuecomment-1457761017
I don't think this verbosity is needed, as the API is already available as Javadocs (there's no additional information that is not also available as Javadocs). FYI, the package private classes need to be part of the apijar, as they are referred to. An alternative would be to remove those from the implements/superclasses, but thats to risky.
I think we should place a note in our NOTICE.txt with Licensing and/or leave a README in the generated files folder.
What do you think? @dweiss
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on a diff in pull request #12188: Alternative version: Implement MMapDirectory with Java 19/20 Project Panama Preview API
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on code in PR #12188:
URL: https://github.com/apache/lucene/pull/12188#discussion_r1127459715
##########
gradle/generation/panama-foreign.gradle:
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+def resources = scriptResources(buildscript)
+
+configure(project(":lucene:core")) {
+ ext {
+ apijars = file('src/generated/jdk');
+ panamaJavaVersions = [ 19, 20 ]
+ }
+
+ configurations {
+ apiextractor
+ }
+
+ dependencies {
+ apiextractor "org.ow2.asm:asm:${scriptDepVersions['asm']}"
+ }
+
+ for (jdkVersion : panamaJavaVersions) {
+ def task = tasks.create(name: "generatePanamaForeignApiJar${jdkVersion}", type: JavaExec) {
+ description "Regenerate the API-only JAR file with public Panama Foreign API from JDK ${jdkVersion}"
+ group "generation"
+
+ javaLauncher = javaToolchains.launcherFor {
+ languageVersion = JavaLanguageVersion.of(jdkVersion)
+ }
+
+ onlyIf {
+ try {
+ javaLauncher.get()
+ return true
+ } catch (Exception e) {
+ logger.warn('Launcher for Java {} is not available; skipping regeneration of Panama Foreign API JAR.', jdkVersion)
+ logger.warn('Error: {}', e.cause?.message)
+ logger.warn("Please make sure to point env 'JAVA{}_HOME' to exactly JDK version {} or enable Gradle toolchain auto-download.", jdkVersion, jdkVersion)
+ return false
Review Comment:
I added this eplicit to not fail on "regenerate". Actually regenerating is not needed, as it will normally produce always the same apijar files, unless JDK changes. The regenerate task here is more for "legal" or "update" reasons, so anybody can regenerate those apijar files if needed - under normal circumstances thats not needed.
This is why I added this check. One reason is also that JDK 20 is not yet released, so the autodownloader does not work. We can remove that piece of code, but it works and complains loud that it did not do anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #12188: Alternative version: Implement MMapDirectory with Java 19/20 Project Panama Preview API
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #12188:
URL: https://github.com/apache/lucene/pull/12188#issuecomment-1456441609
> In this case, would it make sense to remove the logging message warning us that we don't have JDK19 so we're missing out on the goodness? (Currently I have hacked my logger to suppress - wonder if we need to encourage people to upgrade their JDK though? Maybe I'm misunderstanding the purpose).
You mean on Java 20 or 21 at moment? Yes we can think about removing that log message on startup but thats unrelated to this issue. This is about building Lucene.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #12188: Alternative version: Implement MMapDirectory with Java 19/20 Project Panama Preview API
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #12188:
URL: https://github.com/apache/lucene/pull/12188#issuecomment-1457739678
This is my local stashed patch (I used it to review the output):
```patch
gradle/generation/panama-foreign.gradle | 6 ++++-
.../panama-foreign/ExtractForeignAPI.java | 29 ++++++++++++++--------
2 files changed, 24 insertions(+), 11 deletions(-)
diff --git a/gradle/generation/panama-foreign.gradle b/gradle/generation/panama-foreign.gradle
index 694c4656e2f..a4038961ea0 100644
--- a/gradle/generation/panama-foreign.gradle
+++ b/gradle/generation/panama-foreign.gradle
@@ -28,7 +28,7 @@ configure(project(":lucene:core")) {
}
dependencies {
- apiextractor "org.ow2.asm:asm:${scriptDepVersions['asm']}"
+ apiextractor "org.ow2.asm:asm-util:${scriptDepVersions['asm']}"
}
for (jdkVersion : panamaJavaVersions) {
@@ -54,9 +54,13 @@ configure(project(":lucene:core")) {
classpath = configurations.apiextractor
mainClass = file("${resources}/ExtractForeignAPI.java") as String
+ systemProperties = [
+ 'line.separator': "\n",
+ ]
args = [
jdkVersion,
new File(apijars, "panama-foreign-jdk${jdkVersion}.apijar"),
+ new File(apijars, "panama-foreign-jdk${jdkVersion}.txt"),
]
}
diff --git a/gradle/generation/panama-foreign/ExtractForeignAPI.java b/gradle/generation/panama-foreign/ExtractForeignAPI.java
index 3eecbe56149..944b4922f47 100644
--- a/gradle/generation/panama-foreign/ExtractForeignAPI.java
+++ b/gradle/generation/panama-foreign/ExtractForeignAPI.java
@@ -15,6 +15,7 @@
* limitations under the License.
*/
import java.io.IOException;
+import java.io.PrintWriter;
import java.net.URI;
import java.nio.file.Files;
import java.nio.file.Path;
@@ -34,32 +35,40 @@ import org.objectweb.asm.FieldVisitor;
import org.objectweb.asm.MethodVisitor;
import org.objectweb.asm.Opcodes;
import org.objectweb.asm.Type;
+import org.objectweb.asm.util.TraceClassVisitor;
public final class ExtractForeignAPI {
private static final FileTime FIXED_FILEDATE = FileTime.from(Instant.parse("2022-01-01T00:00:00Z"));
public static void main(String... args) throws IOException {
- if (args.length != 2) {
- throw new IllegalArgumentException("Need two parameters: java version, output file");
+ if (args.length != 3) {
+ throw new IllegalArgumentException("Need three parameters: java version, apijar output file, text output file");
}
+ var apijarPath = Paths.get(args[0]);
if (Integer.parseInt(args[0]) != Runtime.version().feature()) {
throw new IllegalStateException("Incorrect java version: " + Runtime.version().feature());
}
var outputPath = Paths.get(args[1]);
+ var signaturesPath = Paths.get(args[2]);
var javaBaseModule = Paths.get(URI.create("jrt:/")).resolve("java.base").toRealPath();
var fileMatcher = javaBaseModule.getFileSystem().getPathMatcher("glob:java/{lang/foreign/*,nio/channels/FileChannel}.class");
- try (var out = new ZipOutputStream(Files.newOutputStream(outputPath)); var stream = Files.walk(javaBaseModule)) {
- var filesToExtract = stream.map(javaBaseModule::relativize).filter(fileMatcher::matches).sorted().collect(Collectors.toList());
+ try (var apijar = new ZipOutputStream(Files.newOutputStream(apijarPath));
+ var signatures = new PrintWriter(Files.newBufferedWriter(signaturesPath));
+ var classes = Files.walk(javaBaseModule)) {
+ signatures.println("// These API signatures of Panama Foreign API were extracted from JDK " + Runtime.version().feature());
+ signatures.println("// Do not modify this file or the corresponding .apijar file which is used for MR-JAR compilation!");
+ var filesToExtract = classes.map(javaBaseModule::relativize).filter(fileMatcher::matches).sorted().collect(Collectors.toList());
for (Path relative : filesToExtract) {
+ signatures.println();
System.out.println("Processing class file: " + relative);
try (var in = Files.newInputStream(javaBaseModule.resolve(relative))) {
final var reader = new ClassReader(in);
final var cw = new ClassWriter(0);
- reader.accept(new Cleaner(cw), ClassReader.SKIP_CODE | ClassReader.SKIP_DEBUG | ClassReader.SKIP_FRAMES);
- out.putNextEntry(new ZipEntry(relative.toString()).setLastModifiedTime(FIXED_FILEDATE));
- out.write(cw.toByteArray());
- out.closeEntry();
+ reader.accept(new Cleaner(cw, signatures), ClassReader.SKIP_CODE | ClassReader.SKIP_DEBUG | ClassReader.SKIP_FRAMES);
+ apijar.putNextEntry(new ZipEntry(relative.toString()).setLastModifiedTime(FIXED_FILEDATE));
+ apijar.write(cw.toByteArray());
+ apijar.closeEntry();
}
}
}
@@ -69,8 +78,8 @@ public final class ExtractForeignAPI {
private static final String PREVIEW_ANN = "jdk/internal/javac/PreviewFeature";
private static final String PREVIEW_ANN_DESCR = Type.getObjectType(PREVIEW_ANN).getDescriptor();
- Cleaner(ClassWriter out) {
- super(Opcodes.ASM9, out);
+ Cleaner(ClassWriter out, PrintWriter writer) {
+ super(Opcodes.ASM9, new TraceClassVisitor(out, writer));
}
private static boolean isHidden(int access) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #12188: Alternative version: Implement MMapDirectory with Java 19/20 Project Panama Preview API
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #12188:
URL: https://github.com/apache/lucene/pull/12188#issuecomment-1460099066
FYI,
I also checked this on the branch_9x with Java 11. It works the same way, the only issue is that the apijar files need to contain also signatures of `java.util.Objects`, because `checkIndex()` with `long` arguments is missing in Java 11, so to actually compile with long indexes we need that class in the module patch, too. But the compiler complains easy, so it was easy to fix.
So I think this is ready to go into main and 9.x. I will merge this till end of week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #12188: Alternative version: Implement MMapDirectory with Java 19/20 Project Panama Preview API
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #12188:
URL: https://github.com/apache/lucene/pull/12188#issuecomment-1457696174
> I like it, Uwe. I have no idea about legal implications (whether interfaces alone are something that can be copied over).
I had the same mental problem already when I released forbiddenapis in 2012. The deprectaed signatures were generated by scanning through rt.jar/module system and adding the signatures to file, e.g, for Java 17: https://github.com/policeman-tools/forbidden-apis/blob/main/src/main/resources/de/thetaphi/forbiddenapis/signatures/jdk-deprecated-17.txt
The only difference here is that the file is binary. As said before I have a code path here that uses ASM's Textifier to create a text file with all signatures in a similar way like forbiddenapis. We could also add a README.txt in the generated files folder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] dweiss commented on pull request #12188: Alternative version: Implement MMapDirectory with Java 19/20 Project Panama Preview API
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #12188:
URL: https://github.com/apache/lucene/pull/12188#issuecomment-1457853933
I'm not a lawyer but I think a note in README next to those apijars is sufficient. There's no wrongdoing here and it's not a critical part of Lucene. I don't see how we could infringe anybody's rights here by extracting those interfaces just to ease the pain of compilation/ building Lucene.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] dweiss commented on a diff in pull request #12188: Alternative version: Implement MMapDirectory with Java 19/20 Project Panama Preview API
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on code in PR #12188:
URL: https://github.com/apache/lucene/pull/12188#discussion_r1127376917
##########
gradle/generation/panama-foreign.gradle:
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+def resources = scriptResources(buildscript)
+
+configure(project(":lucene:core")) {
+ ext {
+ apijars = file('src/generated/jdk');
+ panamaJavaVersions = [ 19, 20 ]
+ }
+
+ configurations {
+ apiextractor
+ }
+
+ dependencies {
+ apiextractor "org.ow2.asm:asm:${scriptDepVersions['asm']}"
+ }
+
+ for (jdkVersion : panamaJavaVersions) {
+ def task = tasks.create(name: "generatePanamaForeignApiJar${jdkVersion}", type: JavaExec) {
+ description "Regenerate the API-only JAR file with public Panama Foreign API from JDK ${jdkVersion}"
+ group "generation"
+
+ javaLauncher = javaToolchains.launcherFor {
+ languageVersion = JavaLanguageVersion.of(jdkVersion)
+ }
+
+ onlyIf {
+ try {
+ javaLauncher.get()
+ return true
+ } catch (Exception e) {
+ logger.warn('Launcher for Java {} is not available; skipping regeneration of Panama Foreign API JAR.', jdkVersion)
+ logger.warn('Error: {}', e.cause?.message)
+ logger.warn("Please make sure to point env 'JAVA{}_HOME' to exactly JDK version {} or enable Gradle toolchain auto-download.", jdkVersion, jdkVersion)
+ return false
Review Comment:
Isn't this going to fail earlier (gradle failing to download the specified toolchain)? I'm surprised this is needed. And if it does return the toolchain, then if it fails, shouldn't it stop the build (not just warn)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler merged pull request #12188: Alternative version: Implement MMapDirectory with Java 19/20 Project Panama Preview API
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler merged PR #12188:
URL: https://github.com/apache/lucene/pull/12188
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #12188: Alternative version: Implement MMapDirectory with Java 19/20 Project Panama Preview API
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #12188:
URL: https://github.com/apache/lucene/pull/12188#issuecomment-1456071249
> > The same approach could be used for vector APIs in future. Please add a comment about if you agree to do it like that!
>
> when the vector api finally comes out, don't interrupt my shuffleboard match to tell me about it!
It has not even reached preview phase, so you can game without stopping for the next 10 years, for sure!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #12188: Alternative version: Implement MMapDirectory with Java 19/20 Project Panama Preview API
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #12188:
URL: https://github.com/apache/lucene/pull/12188#issuecomment-1456118157
I moved the signatures files to src/generated/jdk folder. The lib folder was just a quick hack at beginning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] dweiss commented on pull request #12188: Alternative version: Implement MMapDirectory with Java 19/20 Project Panama Preview API
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #12188:
URL: https://github.com/apache/lucene/pull/12188#issuecomment-1456065664
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] msokolov commented on pull request #12188: Alternative version: Implement MMapDirectory with Java 19/20 Project Panama Preview API
Posted by "msokolov (via GitHub)" <gi...@apache.org>.
msokolov commented on PR #12188:
URL: https://github.com/apache/lucene/pull/12188#issuecomment-1456410971
In this case, would it make sense to remove the logging message warning us that we don't have JDK19 so we're missing out on the goodness? (Currently I have hacked my logger to suppress - wonder if we need to encourage people to upgrade their JDK though? Maybe I'm misunderstanding the purpose).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org