You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2021/06/07 12:06:41 UTC
[GitHub] [lucene] uschindler opened a new pull request #173: Initial rewrite of MMapDirectory for JDK-16 preview (incubating) Panama APIs (>= JDK-16-ea-b32)
uschindler opened a new pull request #173:
URL: https://github.com/apache/lucene/pull/173
**INFO: This is a clone/update of https://github.com/apache/lucene-solr/pull/2176** _(for more detailed discussion see this old PR from the Lucene/Solr combined repository)_
This is just a draft PR for a first insight on memory mapping improvements in JDK 16+.
Some background information: Starting with JDK-14, there is a new incubating module "jdk.incubator.foreign" that has a new, not yet stable API for accessing off-heap memory (and later it will also support calling functions using classical MethodHandles that are located in libraries like .so or .dll files). This incubator module has several versions:
- first version: https://openjdk.java.net/jeps/370 (slow, very buggy and thread confinement, so making it unuseable with Lucene)
- second version: https://openjdk.java.net/jeps/383 (still thread confinement, but now allows transfer of "ownership" to other threads; this is still impossible to use with Lucene.
- third version in JDK 16: https://openjdk.java.net/jeps/393 (this version has included "Support for shared segments"). This now allows us to safely use the same external mmaped memory from different threads and also unmap it!
This module more or less overcomes several problems:
- ByteBuffer API is limited to 32bit (in fact MMapDirectory has to chunk in 1 GiB portions)
- There is no official way to unmap ByteBuffers when the file is no longer used. There is a way to use `sun.misc.Unsafe` and forcefully unmap segments, but any IndexInput accessing the file from another thread will crush the JVM with SIGSEGV or SIGBUS. We learned to live with that and we happily apply the unsafe unmapping, but that's the main issue.
@uschindler had many discussions with the team at OpenJDK and finally with the third incubator, we have an API that works with Lucene. It was very fruitful discussions (thanks to @mcimadamore !)
With the third incubator we are now finally able to do some tests (especially performance). As this is an incubating module, this PR first changes a bit the build system:
- disable `-Werror` for `:lucene:core`
- add the incubating module to compiler of `:lucene:core` and enable it for all test builds. This is important, as you have to pass `--add-modules jdk.incubator.foreign` also at runtime!
The code basically just modifies `MMapDirectory` to use LONG instead of INT for the chunk size parameter. In addition it adds `MemorySegmentIndexInput` that is a copy of our `ByteBufferIndexInput` (still there, but unused), but using MemorySegment instead of ByteBuffer behind the scenes. It works in exactly the same way, just the try/catch blocks for supporting EOFException or moving to another segment were rewritten.
The openInput code uses `MemorySegment.mapFile()` to get a memory mapping. This method is unfortunately a bit buggy in JDK-16-ea-b30, so I added some workarounds. See JDK issues: https://bugs.openjdk.java.net/browse/JDK-8259027, https://bugs.openjdk.java.net/browse/JDK-8259028, https://bugs.openjdk.java.net/browse/JDK-8259032, https://bugs.openjdk.java.net/browse/JDK-8259034. The bugs with alignment and zero byte mmaps are fixed in b32, this PR was adapted (hacks removed).
It passes all tests and it looks like you can use it to read indexes. The default chunk size is now 16 GiB (but you can raise or lower it as you like; tests are doing this). Of course you can set it to Long.MAX_VALUE, in that case every index file is always mapped to one big memory mapping. My testing with Windows 10 have shown, that this is *not a good idea!!!*. Huge mappings fragment address space over time and as we can only use like 43 or 46 bits (depending on OS), the fragmentation will at some point kill you. So 16 GiB looks like a good compromise: Most files will be smaller than 6 GiB anyways (unless you optimize your index to one huge segment). So for most Lucene installations, the number of segments will equal the number of open files, so Elasticsearch huge user consumers will be very happy. The sysctl max_map_count may not need to be touched anymore.
In addition, this implements `readLELongs` in a better way than @jpountz did (no caching or arbitrary objects). Nevertheless, as the new MemorySegment API relies on final, unmodifiable classes and coping memory from a MemorySegment to a on-heap Java array, it requires us to wrap all those arrays using a MemorySegment each time (e.g. in `readBytes()` or `readLELongs`), there may be some overhead du to short living object allocations (those are NOT reuseable!!!). _In short: In future we should throw away on coping/loading our stuff to heap and maybe throw away IndexInput completely and base our code fully on random access. The new foreign-vector APIs will in future also be written with MemorySegment in its focus. So you can allocate a vector view on a MemorySegment and let the vectorizer fully work outside java heap inside our mmapped files! :-)_
It would be good if you could checkout this branch and try it in production.
But be aware:
- You need JDK 11 to run Gradle (set `JAVA_HOME` to it)
- You need JDK 16-ea-b32 (set `RUNTIME_JAVA_HOME` to it)
- The lucene-core.jar will be JDK16 class files and requires JDK-16 to execute.
- Also you need to add `--add-modules jdk.incubator.foreign` to the command line of your Java program/Solr server/Elasticsearch server
It would be good to get some benchmarks, especially by @rmuir or @mikemccand. _Take your time and enjoy the complexity of setting this up!_ ;-)
My plan is the following:
- report any bugs or slowness, especially with Hotspot optimizations. The last time I talked to Maurizio, he taked about Hotspot not being able to fully optimize for-loops with long instead of int, so it may take some time until the full performance is there.
- wait until the final version of project PANAMA-foreign goes into Java's Core Library (no module needed anymore)
- add a MR-JAR for lucene-core.jar and compile the MemorySegmentIndexInput and maybe some helper classes with JDK 17/18/19 (hopefully?).
~~In addition there are some comments in the code talking about safety (e.g., we need `IOUtils.close()` taking `AutoCloseable` instead of just `Closeable`, so we can also enfoce that all memory segments are closed after usage.~~ In addition, by default all VarHandles are aligned. By default it refuses to read a LONG from an address which is not a multiple of 8. I had to disable this feature, as all our index files are heavily unaliged. We should in meantime not only convert our files to little endian, but also make all non-compressed types (like `long[]` arrays or non-encoded integers be aligned to the correct boundaries in files). The most horrible thing I have seen is that our CFS file format starts the "inner" files totally unaligned. We should fix the CFSWriter to start new files always at multiples of 8 bytes. I will open an issue about this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on a change in pull request #173: Initial rewrite of MMapDirectory for JDK-16 preview (incubating) Panama APIs (>= JDK-16-ea-b32)
Posted by GitBox <gi...@apache.org>.
uschindler commented on a change in pull request #173:
URL: https://github.com/apache/lucene/pull/173#discussion_r646596993
##########
File path: lucene/test-framework/src/java/org/apache/lucene/mockfile/FilterPath.java
##########
@@ -261,10 +267,7 @@ public boolean equals(Object obj) {
* @return innermost Path instance
*/
public static Path unwrap(Path path) {
Review comment:
That's how it is used: https://github.com/apache/lucene/pull/173/files#diff-efeb36d227b58a0ad42ddf35d0e260ca8010021f8dbc78627333085defa16a13R251
You pass in a `java.nio.files.Path` and it gets unwrapped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] dweiss commented on a change in pull request #173: Initial rewrite of MMapDirectory for JDK-16 preview (incubating) Panama APIs (>= JDK-16-ea-b32)
Posted by GitBox <gi...@apache.org>.
dweiss commented on a change in pull request #173:
URL: https://github.com/apache/lucene/pull/173#discussion_r646570141
##########
File path: lucene/test-framework/src/java/org/apache/lucene/mockfile/FilterPath.java
##########
@@ -261,10 +267,7 @@ public boolean equals(Object obj) {
* @return innermost Path instance
*/
public static Path unwrap(Path path) {
Review comment:
I thought you might wanted to pass arbitrary objects there... Not much can be done then, sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] dweiss commented on a change in pull request #173: Initial rewrite of MMapDirectory for JDK-16 preview (incubating) Panama APIs (>= JDK-16-ea-b32)
Posted by GitBox <gi...@apache.org>.
dweiss commented on a change in pull request #173:
URL: https://github.com/apache/lucene/pull/173#discussion_r646540797
##########
File path: lucene/core/src/java/org/apache/lucene/util/Unwrapable.java
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.util;
+
+/**
+ * An object with this interface is a wrapper around another object
+ * (e.g., a filter with a delegate). The method {@link #unwrap()} can
+ * be called to get the wrapped object
+ *
+ * @lucene.internal
+ */
+public interface Unwrapable<T> {
+
+ /** Unwraps this instance */
+ T unwrap();
+
+ /**
+ * Unwraps all {@code Unwrapable}s around the given object.
+ */
+ @SuppressWarnings("unchecked")
+ public static <T> T unwrapAll(T o) {
Review comment:
Make the generic definition as <T extends Unwrapable> so that no class checks are needed and the compiler can verify if a given type implements Unwrapable?
##########
File path: lucene/test-framework/src/java/org/apache/lucene/mockfile/FilterPath.java
##########
@@ -261,10 +267,7 @@ public boolean equals(Object obj) {
* @return innermost Path instance
*/
public static Path unwrap(Path path) {
Review comment:
Maybe it should just be a default impl. on the interface itself... Then no need for multiple declarations and you'd know the type right away (vide my previous comment).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] xiaoshi2013 commented on pull request #173: Initial rewrite of MMapDirectory for JDK-16 preview (incubating) Panama APIs (>= JDK-16-ea-b32)
Posted by GitBox <gi...@apache.org>.
xiaoshi2013 commented on pull request #173:
URL: https://github.com/apache/lucene/pull/173#issuecomment-915366162
so cool!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #173: Initial rewrite of MMapDirectory for JDK-16 preview (incubating) Panama APIs (>= JDK-16-ea-b32)
Posted by GitBox <gi...@apache.org>.
uschindler commented on pull request #173:
URL: https://github.com/apache/lucene/pull/173#issuecomment-855892370
I invite everybody to attend this talk: https://2021.berlinbuzzwords.de/session/future-lucenes-mmapdirectory-why-use-it-and-whats-coming-java-16-and-later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler closed pull request #173: Initial rewrite of MMapDirectory for JDK-16 preview (incubating) Panama APIs (>= JDK-16-ea-b32)
Posted by GitBox <gi...@apache.org>.
uschindler closed pull request #173:
URL: https://github.com/apache/lucene/pull/173
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #173: Initial rewrite of MMapDirectory for JDK-16 preview (incubating) Panama APIs (>= JDK-16-ea-b32)
Posted by GitBox <gi...@apache.org>.
uschindler commented on pull request #173:
URL: https://github.com/apache/lucene/pull/173#issuecomment-859613146
> Is the plan to work on this and #177 in parallel until we know which is the more sustainable option, or abandon this one altogether with expectations that JDK 17 will be better? I'm going to go through the pain of setting one up but probably cannot do both.
I won't expect any of both to be in a stable release of Lucene yet. The version for 16 (this one) is here for reference only. It has performance problems, because JDK 16 was not able to optimize loops with 64 bit. With JDK 17, this should be better, but I wasn't able to test this with #177. So I'd recommend to do performance tests with #177.
Hopefully at some point this will land in JDK so we can officially use it. The problem is currently, every major release changes API in significant ways, so there is no way to include it in official builds. It also won't work without command line settings, so if you'd like to use with software like Solr or Elasticsearch, you would need to change startup scripts, too. MR-JARs don't help, because MR-JARs resolve class files based on minimal version and you can't make it "only use this class for JDK 16".
A plan might be (as this is quite isolated) to create a separate github project, with just the directory implementation, so it can be downloaded as separate JAR file and included into projects. Possibly with a DirectoryFactory for Solr or similar plugin for Elasticsearch. My time is a bit limited at moment, but that's obviously the best way to go. The setup as draft pull request with hacked code inside Lucene was mainly done to run all Lucene tests easily against it and compare performance with old MMapDirectory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #173: Initial rewrite of MMapDirectory for JDK-16 preview (incubating) Panama APIs (>= JDK-16-ea-b32)
Posted by GitBox <gi...@apache.org>.
uschindler commented on pull request #173:
URL: https://github.com/apache/lucene/pull/173#issuecomment-856594835
Hi,
I am currently working on an Update to Java 17, where some changes occurred (there is a new `ResourceScope` class that replaces `MemorySegment.share()`; the good thing: You can close only the `ResourceScope`, not segments anymore. And: you can close the ResourceScope and all segments attached to it are closed. In short: `MemorySegmentIndexInput.close()` simply closes the `ResourceScope` -- done).
I will open a new JDK17 PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on a change in pull request #173: Initial rewrite of MMapDirectory for JDK-16 preview (incubating) Panama APIs (>= JDK-16-ea-b32)
Posted by GitBox <gi...@apache.org>.
uschindler commented on a change in pull request #173:
URL: https://github.com/apache/lucene/pull/173#discussion_r646550354
##########
File path: lucene/test-framework/src/java/org/apache/lucene/mockfile/FilterPath.java
##########
@@ -261,10 +267,7 @@ public boolean equals(Object obj) {
* @return innermost Path instance
*/
public static Path unwrap(Path path) {
Review comment:
...but wont work: The method here is obsolete, I just kept it inside to not touch any test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on a change in pull request #173: Initial rewrite of MMapDirectory for JDK-16 preview (incubating) Panama APIs (>= JDK-16-ea-b32)
Posted by GitBox <gi...@apache.org>.
uschindler commented on a change in pull request #173:
URL: https://github.com/apache/lucene/pull/173#discussion_r646547379
##########
File path: lucene/core/src/java/org/apache/lucene/util/Unwrapable.java
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.util;
+
+/**
+ * An object with this interface is a wrapper around another object
+ * (e.g., a filter with a delegate). The method {@link #unwrap()} can
+ * be called to get the wrapped object
+ *
+ * @lucene.internal
+ */
+public interface Unwrapable<T> {
+
+ /** Unwraps this instance */
+ T unwrap();
+
+ /**
+ * Unwraps all {@code Unwrapable}s around the given object.
+ */
+ @SuppressWarnings("unchecked")
+ public static <T> T unwrapAll(T o) {
Review comment:
Thanks, this was some funny hack I applied after I got crazy to find a generic solution. What do you think: Maybe we should make a separate issue out of Unwrapable ?
The problem here and why I did not add generics for the static method: Not all implementations you can pass in may implement unwrapable. java.nio.file.Path does not implement unwrappable and the idea of the static method is to pass in any Path and it will unwrap, if possible. This is a static method, so you can use it with any Path! If the path is not unwrappable, it would do nothing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on a change in pull request #173: Initial rewrite of MMapDirectory for JDK-16 preview (incubating) Panama APIs (>= JDK-16-ea-b32)
Posted by GitBox <gi...@apache.org>.
uschindler commented on a change in pull request #173:
URL: https://github.com/apache/lucene/pull/173#discussion_r646547557
##########
File path: lucene/test-framework/src/java/org/apache/lucene/mockfile/FilterPath.java
##########
@@ -261,10 +267,7 @@ public boolean equals(Object obj) {
* @return innermost Path instance
*/
public static Path unwrap(Path path) {
Review comment:
That's a fair point. :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] dweiss commented on a change in pull request #173: Initial rewrite of MMapDirectory for JDK-16 preview (incubating) Panama APIs (>= JDK-16-ea-b32)
Posted by GitBox <gi...@apache.org>.
dweiss commented on a change in pull request #173:
URL: https://github.com/apache/lucene/pull/173#discussion_r646570141
##########
File path: lucene/test-framework/src/java/org/apache/lucene/mockfile/FilterPath.java
##########
@@ -261,10 +267,7 @@ public boolean equals(Object obj) {
* @return innermost Path instance
*/
public static Path unwrap(Path path) {
Review comment:
I thought you may want to pass arbitrary objects there... Not much can be done then, sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #173: Initial rewrite of MMapDirectory for JDK-16 preview (incubating) Panama APIs (>= JDK-16-ea-b32)
Posted by GitBox <gi...@apache.org>.
uschindler commented on pull request #173:
URL: https://github.com/apache/lucene/pull/173#issuecomment-1039342764
This PR is no longer maintained!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #173: Initial rewrite of MMapDirectory for JDK-16 preview (incubating) Panama APIs (>= JDK-16-ea-b32)
Posted by GitBox <gi...@apache.org>.
uschindler commented on pull request #173:
URL: https://github.com/apache/lucene/pull/173#issuecomment-856682232
The JDK 17 version is now here: #177
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #173: Initial rewrite of MMapDirectory for JDK-16 preview (incubating) Panama APIs (>= JDK-16-ea-b32)
Posted by GitBox <gi...@apache.org>.
uschindler commented on pull request #173:
URL: https://github.com/apache/lucene/pull/173#issuecomment-855903431
On the Windown build I have seen this error on Jenkins:
```
1> 2.857s 1.5s: idle P0 [ set repls] top: set replicasIDs=[1] tcpPorts=[55991]
1> 2.888s 1.5s: idle P0 [ indexing] start handling indexing socket=Socket[addr=/127.0.0.1,port=55995,localport=55988]
1> 3.099s 1.7s: idle P0 [ flush] now flush; 1 replicas
1> 3.100s 1.7s: idle P0 [ flush] top: now flushAndRefresh
1> 3.127s 1.7s: idle P0 [ flush] unexpected exception handling client connection; now failing test:
1> 3.130s 1.5s: parent [ pump0] java.lang.NoClassDefFoundError: jdk/incubator/foreign/MemorySegment
1> 3.131s 1.5s: parent [ pump0] at org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:252)
1> 3.132s 1.5s: parent [ pump0] at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:232)
1> 3.133s 1.5s: parent [ pump0] at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:3053)
1> 3.134s 1.5s: parent [ pump0] at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:793)
1> 3.134s 1.5s: parent [ pump0] at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101)
1> 3.134s 1.5s: parent [ pump0] at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101)
1> 3.134s 1.5s: parent [ pump0] at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:152)
1> 3.134s 1.5s: parent [ pump0] at org.apache.lucene.codecs.lucene90.compressing.FieldsIndexWriter.finish(FieldsIndexWriter.java:124)
1> 3.135s 1.5s: parent [ pump0] at org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.finish(Lucene90CompressingStoredFieldsWriter.java:491)
1> 3.135s 1.5s: parent [ pump0] at org.apache.lucene.index.StoredFieldsConsumer.flush(StoredFieldsConsumer.java:82)
1> 3.135s 1.5s: parent [ pump0] at org.apache.lucene.index.IndexingChain.flush(IndexingChain.java:270)
1> 3.137s 1.5s: parent [ pump0] at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:386)
1> 3.137s 1.5s: parent [ pump0] at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:497)
1> 3.137s 1.5s: parent [ pump0] at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:676)
1> 3.138s 1.5s: parent [ pump0] at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:568)
1> 3.138s 1.5s: parent [ pump0] at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:380)
1> 3.138s 1.5s: parent [ pump0] at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:354)
1> 3.138s 1.5s: parent [ pump0] at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:344)
1> 3.138s 1.5s: parent [ pump0] at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:163)
1> 3.139s 1.5s: parent [ pump0] at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:144)
1> 3.139s 1.5s: parent [ pump0] at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:52)
1> 3.139s 1.5s: parent [ pump0] at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167)
1> 3.139s 1.5s: parent [ pump0] at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240)
1> 3.139s 1.5s: parent [ pump0] at org.apache.lucene.replicator.nrt.PrimaryNode.flushAndRefresh(PrimaryNode.java:164)
1> 3.140s 1.5s: parent [ pump0] at org.apache.lucene.replicator.nrt.SimplePrimaryNode.handleFlush(SimplePrimaryNode.java:345)
1> 3.140s 1.5s: parent [ pump0] at org.apache.lucene.replicator.nrt.SimplePrimaryNode.handleOneConnection(SimplePrimaryNode.java:688)
1> 3.140s 1.5s: parent [ pump0] at org.apache.lucene.replicator.nrt.TestSimpleServer$ClientHandler.run(TestSimpleServer.java:98)
1> 3.140s 1.5s: parent [ pump0] Caused by: java.lang.ClassNotFoundException: jdk.incubator.foreign.MemorySegment
1> 3.140s 1.5s: parent [ pump0] at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:636)
1> 3.140s 1.5s: parent [ pump0] at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:182)
1> 3.142s 1.5s: parent [ pump0] at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:519)
1> 3.142s 1.5s: parent [ pump0] ... 27 more
1> 3.130s 1.7s: idle P0 [main child 0] top: server socket exc; now exit
1> 3.131s 1.7s: idle P0 [main child 0] top: join clientThread=Thread[flush,5,TGRP-TestSimpleServer]
> java.io.EOFException
> at __randomizedtesting.SeedInfo.seed([96E503E953A21F6D:D20A30ADDC1FD0FE]:0)
> at org.apache.lucene.store.InputStreamDataInput.readByte(InputStreamDataInput.java:32)
> at org.apache.lucene.store.DataInput.readInt(DataInput.java:104)
> at org.apache.lucene.store.DataInput.readLong(DataInput.java:166)
> at org.apache.lucene.replicator.nrt.NodeProcess.flush(NodeProcess.java:140)
> at org.apache.lucene.replicator.nrt.TestNRTReplication.testCrashReplica(TestNRTReplication.java:766)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78)
> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:567)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
> at java.base/java.lang.Thread.run(Thread.java:831)
2> NOTE: reproduce with: gradlew test --tests TestNRTReplication.testCrashReplica -Dtests.seed=96E503E953A21F6D -Dtests.slow=true -Dtests.directory=MMapDirectory -Dtests.locale=fr-SC -Dtests.timezone=Etc/GMT+10 -Dtests.asserts=true -Dtests.file.encoding=windows-1252
```
Looks like the NRT test does not pass the full JVM command line. I won't change this in the PR, but add some workaround.
Interestingly it did not fail on my local Windows and also not on Linux Jenkins. So it seems to be that the parent test does not even pass all system properties to the underlying test, so it uses directory randomization, although Jenkins hardcoded `-Dtests.directory=MMapDirectory`. This is why it does not fail consistently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #173: Initial rewrite of MMapDirectory for JDK-16 preview (incubating) Panama APIs (>= JDK-16-ea-b32)
Posted by GitBox <gi...@apache.org>.
uschindler commented on pull request #173:
URL: https://github.com/apache/lucene/pull/173#issuecomment-1039342764
This PR is no longer maintained!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler closed pull request #173: Initial rewrite of MMapDirectory for JDK-16 preview (incubating) Panama APIs (>= JDK-16-ea-b32)
Posted by GitBox <gi...@apache.org>.
uschindler closed pull request #173:
URL: https://github.com/apache/lucene/pull/173
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] MarcusSorealheis commented on pull request #173: Initial rewrite of MMapDirectory for JDK-16 preview (incubating) Panama APIs (>= JDK-16-ea-b32)
Posted by GitBox <gi...@apache.org>.
MarcusSorealheis commented on pull request #173:
URL: https://github.com/apache/lucene/pull/173#issuecomment-859080425
I know this is early days but want to clarify:
Is the plan to work on this and #177 in parallel until we know which is the more sustainable option, or abandon this one altogether with expectations that JDK 17 will be better? I'm going to go through the pain of setting one up but probably cannot do both.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #173: Initial rewrite of MMapDirectory for JDK-16 preview (incubating) Panama APIs (>= JDK-16-ea-b32)
Posted by GitBox <gi...@apache.org>.
uschindler commented on pull request #173:
URL: https://github.com/apache/lucene/pull/173#issuecomment-855872206
I moved this old pull request from https://github.com/apache/lucene-solr/pull/2176 to the Lucene repository:
- Removed the changes in Solr
- Updated to Little Endian (see #107, LUCENE-9047)
All tests still pass, the new policeman Jenkins job is: https://jenkins.thetaphi.de/view/Lucene/job/Lucene-jdk16panama-Linux/ (Linux), https://jenkins.thetaphi.de/view/Lucene/job/Lucene-jdk16panama-Windows/ (Windows)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler closed pull request #173: Initial rewrite of MMapDirectory for JDK-16 preview (incubating) Panama APIs (>= JDK-16-ea-b32)
Posted by GitBox <gi...@apache.org>.
uschindler closed pull request #173:
URL: https://github.com/apache/lucene/pull/173
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org