You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "mikemccand (via GitHub)" <gi...@apache.org> on 2025/08/14 11:58:27 UTC

[I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

mikemccand opened a new issue, #15068:
URL: https://github.com/apache/lucene/issues/15068

   ### Description
   
   [I'm opening this for discussion ... I'm not sure we have to fix anything here, but I at least want to document the situation so if other Lucene users hit max maps limit, we can quickly explain why]
   
   At Amazon product search, we've seen our indexer processes sometimes trip the OS hard limit (64 K, though [modern Linuxes (Lini?) have increased to 1 MiB](https://archlinux.org/news/increasing-the-default-vmmax_map_count-value/)) of number of memory-mapped segments, killing the indexing process.  Looking at the maps (`cat /proc/<pid>/maps`) it's clear we are leaking maps for  `_N.si` files, e.g. the exact same `_7.si` file will be mapped 76 times, and same for `_8.si` and all other open segments.
   
   I have a small test case that reproduces this leak (I'll attach it shortly), on 9.12.x, with Java 21 or 24.  It does not reproduce on 10.x because we've [changed `_N.si` files to open with `IOContext.READONCE`](https://github.com/apache/lucene/pull/12027) which [turns off the Arena pooling](https://github.com/apache/lucene/blob/branch_10_2/lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java#L61) (sets `confined=true`).  To repro, you need an index that has at least one segment.  Then you need to hold open a `DirectoryReader` (which creates an Arena for each segment's open files).  Then, periodically, read the latest `segments_N` file, which opens and closes each segment's `_N.si` file, and those `_N.si` maps will be added to the still-open Arenas, not unmap'd until you close your reader or a segment is merged away.
   
   This is not serious for us (Amazon product search) -- we have workaround for 9.12.x (use `NIOFSDirectory`), or, upgrade to 10.x.  Maybe setting `-Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits` to a smallish value (defaults to 1024) would work too, not sure.
   
   Context: we [added this cool Arena pooling](https://github.com/apache/lucene/pull/13570) to Lucene to amortize the sometimes highish JDK cost of unmap (which deopts top frames of all running threads to check that they are not accessing the virtual address space about to be unmap'd), which in turn was discovered by [an upstream benchmark](https://github.com/dacapobench/dacapobench/issues/264) (thank you!!).  https://github.com/apache/lucene/pull/13555 and https://github.com/apache/lucene/issues/13325 have more context.  [JDK-8335480](https://bugs.openjdk.org/browse/JDK-8335480), delivered in JDK 24, tries to reduce the JDK deopt cost of unmap.
   
   Our usage was somewhat expert (periodically reading the latest commit point (`segments_N` file) while indexing), and the leak is fixed in 10.x.  But there is maybe still some open risk if an app uses Lucene APIs to open other files, e.g. maybe the app does lots of deletes against old segments, and maps/unmaps those deletion files, and those "leak"?  Or maybe these paths that reproduce the "leak" are so expert that users won't hit them in practice in 9.x / 10.x?  Or maybe we should decrease the default max maps in a single arena from 1024?
   
   ### Version and environment details
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3188737409

   > I'm not sure why my gradle leaks daemons like a sieve.
   
   Hahaha. 🤓🐰


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "ayinresh (via GitHub)" <gi...@apache.org>.
ayinresh commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3212107301

   Hi @uschindler,
   The fix makes sense and should solve our bug.  Thanks so much for the help!
   
   As for:
   > In addition are you really sure that no other thread that maybe still getting stored fields after all other threads have released the searcher? RefrenceManager only ensures no new threads can aquire a searcher, but nothing prevents other threads from using an open searcher that might have been released!
   
   The answer is no, I'm not sure. It is definitely possible we are doing something incorrect here, and I don't want to mislead the investigation. I will report back if I find solid evidence that there is an issue with `ReferenceManager` itself (and not our mis-use).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3193571970

   > Curiously ChatGPT won't load this conversation ... just a red error box saying "Unable to load conversation....". Are you sure you published / made it public? Sharing is caring.
   
   It should be public, yes. Strange. This is a follow-up - I was curious what it could come up with but it's not any better in the end.
   https://chatgpt.com/share/689edbeb-b298-8011-b031-d2dbcc882e3c
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3212211734

   > Hi [@uschindler](https://github.com/uschindler), The fix makes sense and should solve our bug. Thanks so much for the help!
   > 
   > As for:
   > 
   > > In addition are you really sure that no other thread that maybe still getting stored fields after all other threads have released the searcher? RefrenceManager only ensures no new threads can aquire a searcher, but nothing prevents other threads from using an open searcher that might have been released!
   > 
   > The answer is no, I'm not sure. It is definitely possible we are doing something incorrect here, and I don't want to mislead the investigation. I will report back if I find solid evidence that there is an issue with `ReferenceManager` itself (and not our mis-use).
   
   You could configure MMapDir to disable grouping completely (see above with `dir.setGroupingFunction(MMapDirectory.NO_GROUPING)`). If you then test your code again in production, it should not hang on closing. Instead it might happen that searches running in parallel while you close throw `AlreadyClisedException`. But make sure to have high concurrency.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3191329287

   > > > I'm not sure why my gradle leaks daemons like a sieve.
   > > 
   > > 
   > > Hahaha. 🤓🐰
   > 
   > This is probably because you're using different vms or gradle options (jvm options). I can't say. You (we) have three options - run without the daemon (Robert's choice), lower daemon expiration timeout, write our own build system from scratch. I'm recently leaning towards (3)...
   
   LOL -- I love option 3 -- I've done this too many times in the past!
   
   Thanks @dweiss.  I do jump around with different JVMs, different Lucene branches, etc.  Probably freaks gradle out ...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3188851308

   > I'd strongly recommend to lower the 1024 default. This has shown problematic at many places. IMHO, 64 is much better for pooling.
   
   Agreed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3188470879

   Assuming , JDK 21+ and Lucene 9.12.x:  I have not looked in detail, but `SegmentInfos.readLatestCommit` uses `IOContext.READONCE`, which means that we should use a _confined_ `Arena`, and that arena should be closed which would in turn release the mapping. Clearly something is wrong, but it seems more like a bug in 9.x, since the usage in `readLatestCommit` should result in a confined arena, rather than the more complex shared arena code path.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3188713729

   We should check if there's some general leak in the shared arenas in 9.x.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3188703900

   Can we also look into this one #15054 looks related maybe that explains Opensearch's problem in their RemoteDirectory. It is unclear what they are doing but they also leak files. As they dint explain what they're doing it looks related to some extent. I know it's a different Amazon business but complaints look similar.
   
   I'd strongly recommend to lower the 1024 default. This has shown problematic. IMHO, 64 is much better for pooling.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3214292817

   There's now a PR available: #15112


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3189254182

   @ChrisHegarty: I was reading the code of the RefCountedArena. I still don't understand how this works with the lower and upper 16 bits. I still have the feeling that there is some bug somewhere. I also did not find any test for that code.
   
   I am wondering abount people who told me that they used maxPermits=1 and still run out of mappings... (see #15054).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3191347835

   I tested 64 max permits (the fix in https://github.com/apache/lucene/pull/15078) and we leak for a bit and then the leaks stop, yay!
   
   ```
   ...
   MAP COUNT (from /proc/235240/maps): 928 [420 si files]
   MAP COUNT (from /proc/235240/maps): 938 [430 si files]
   MAP COUNT (from /proc/235240/maps): 948 [440 si files]
   MAP COUNT (from /proc/235240/maps): 958 [450 si files]
   MAP COUNT (from /proc/235240/maps): 968 [460 si files]
   MAP COUNT (from /proc/235240/maps): 978 [470 si files]
   MAP COUNT (from /proc/235240/maps): 988 [480 si files]
   MAP COUNT (from /proc/235240/maps): 998 [490 si files]
   MAP COUNT (from /proc/235240/maps): 1008 [500 si files]
   MAP COUNT (from /proc/235240/maps): 1018 [510 si files]
   MAP COUNT (from /proc/235240/maps): 1028 [520 si files]
   MAP COUNT (from /proc/235240/maps): 1028 [520 si files]
   MAP COUNT (from /proc/235240/maps): 1028 [520 si files]
   MAP COUNT (from /proc/235240/maps): 1028 [520 si files]
   MAP COUNT (from /proc/235240/maps): 1028 [520 si files]
   MAP COUNT (from /proc/235240/maps): 1028 [520 si files]
   MAP COUNT (from /proc/235240/maps): 1028 [520 si files]
   MAP COUNT (from /proc/235240/maps): 1028 [520 si files]
   MAP COUNT (from /proc/235240/maps): 1028 [520 si files]
   MAP COUNT (from /proc/235240/maps): 1028 [520 si files]
   MAP COUNT (from /proc/235240/maps): 1028 [520 si files]
   ```
   
   So this bounds the max maps for any held-open index segment to 64.  Realistically, any app that keeps loading the segments file is presumably doing so because the index is changing and segments are turning over, and then the arenas will be closed, even with leaking.  So the real-world exposure risk seems relatively low, but I'm still curious about https://github.com/apache/lucene/issues/15054.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3190801184

   Ok, I think I get it - the state counts acquire/release permits in both the upper and the lower part of state variable; the distinction is made to differentiate the closed vs nothing-to-release message. Not sure if it's needed or if it could be made simpler.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3189119872

   As a quick workaround for 9.x one could use:
   
   ```java
   mmapDir.setGroupingFunction(fn -> fn.endsWith(".si") ? Optional.empty() : MMapDirectory.GROUP_BY_SEGMENT.apply(fn));
   ```
   
   Uwe


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3190782601

   > We can also discuss this in the other issue. I am still starring at the code and don't get how it works with decrementing. What's the difference between aquires (upper 16 bit) and counts (lower 16 bits), but actually it only decrements the lower 16 bits? Sorry, maybe it's too hot with 35 degrees in Bremen...
   
   it's a balmy 28 here at the Baltic seaside but I also don't get the code in RefCountedSharedArena. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3191326138

   > I am wondering abount people who told me that they used maxPermits=1 and still run out of mappings... (see https://github.com/apache/lucene/issues/15054).
   
   Yeah it's odd it didn't help that issue ... it sidesteps the issue in 9.12.x (before @ChrisHegarty's [fix to switch `_N.si` to `READONCE` so it uses "confined arena"](https://github.com/apache/lucene/pull/15074)).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3196297618

   > > > Curiously ChatGPT won't load this conversation ... just a red error box saying "Unable to load conversation....". Are you sure you published / made it public? Sharing is caring.
   > > 
   > > 
   > > It should be public, yes. Strange. This is a follow-up - I was curious what it could come up with but it's not any better in the end. https://chatgpt.com/share/689edbeb-b298-8011-b031-d2dbcc882e3c
   > 
   > Now it loads fine for me -- very cool interaction and simple prompting! Amazing ChatGPT can understand, and then (sort of) simplify the code.
   
   I think this is mainly caused by the fact that the original code was already following some common pattern (maybe it was created by ChatGPT?). @ChrisHegarty wrote it and while looking for patterns of the code like this, it seems also used commonly INSIDE the JDK, e.g. when reference counting is done otherwise. So it's good that we have a OpenJDK committer who knows all those tricks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3211395643

   Hi @ayinresh, here is the issue opened: #1510
   
   This is areal bug. But actually your code seems to have a bug, too: You seem to close an IndexReader/IndexSearcher while it is still in use by another thread. Only this leads to the fact that the original Arema throws IllegalState on close.
   
   If you have "sane" code your would not be affected by the infinite loop. I think this is custom Lucene code, not Opensearch/Elasticsearch, correct?
   
   Uwe


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3189264266

   > [@ChrisHegarty](https://github.com/ChrisHegarty): I was reading the code of the RefCountedArena. I still don't understand how this works with the lower and upper 16 bits. I still have the feeling that there is some bug somewhere. I also did not find any test for that code.
   > 
   > I am wondering abount people who told me that they used maxPermits=1 and still run out of mappings... (see [#15054](https://github.com/apache/lucene/issues/15054)).
   
   We can also discuss this in the other issue. I am still starring at the code and don't get how it works with decrementing. What's the difference between aquires (upper 16 bit) and counts (lower 16 bits), but actually it only decrements the lower 16 bits? Sorry, maybe it's too hot with 35 degrees in Bremen...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3190741621

   > > I'm not sure why my gradle leaks daemons like a sieve.
   > 
   > Hahaha. 🤓🐰
   
   This is probably because you're using different vms or gradle options (jvm options). I can't say. You (we) have three options - run without the daemon (Robert's choice), lower daemon expiration timeout, write our own build system from scratch. I'm recently leaning towards (3)...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3190794396

   I think we should at least write a unit test just for the ref counted shared arena (it's possible in main easily). The current tests only test mmap dir and check that it does not break, but we do not have no test in main that creates an arena and tests it. A test would also explain how it should work. It may also explain why you need 2 separate reference counters.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3190975704

   The problem of Mike's code is mainly that he calls getSegmetInfos over and over for the same segment. Of course when you open a new IndexInput, it creates a new mapping each time (our refcounted Arena is not able to reuse existing mappings).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3191317050

   > Maybe setting `-Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits` to a smallish value (defaults to 1024) would work too, not sure.
   
   I tested `-Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits=1` and it does indeed prevent the leak with my small test case on 9.12.x, JDK 21:
   
   ```
   MAP COUNT (from /proc/234634/maps): 508 [0 si files]
   MAP COUNT (from /proc/234634/maps): 508 [0 si files]
   MAP COUNT (from /proc/234634/maps): 508 [0 si files]
   MAP COUNT (from /proc/234634/maps): 508 [0 si files]
   MAP COUNT (from /proc/234634/maps): 508 [0 si files]
   MAP COUNT (from /proc/234634/maps): 508 [0 si files]
   MAP COUNT (from /proc/234634/maps): 508 [0 si files]
   MAP COUNT (from /proc/234634/maps): 508 [0 si files]
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3211151520

   I think the code in RefCountedSharedArena should throw another Exception than IllegalStateException, if the refcounting fails to work correctly. We have to verify this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3190818087

   The code seems correct to me (and I ran a simple test to verify). It's more likely that the reference to a RefCountedArena object itself is lost somewhere. This can be debugged with some trickery - add a static Cleaner that tracks gc on the allocated Arenas and makes sure their state count is zeroed/released when arena is gc'ed. If it's not zeroed - it means ref counting failed somewhere. It's a bit convoluted but works - I have code like this used somewhere. I can't work on it today - will be travelling - but if needed, let me know.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3188222106

   This test case repros the leak on 9.12.x:
   
   ```
   import java.io.IOException;
   import java.nio.file.Paths;
   import java.util.concurrent.atomic.AtomicLong;
   import java.util.Random;
   import java.nio.file.Files;
   import java.nio.file.Path;
   
   import org.apache.lucene.analysis.standard.StandardAnalyzer;
   import org.apache.lucene.document.Document;
   import org.apache.lucene.document.Field;
   import org.apache.lucene.document.StringField;
   import org.apache.lucene.document.TextField;
   import org.apache.lucene.index.IndexWriter;
   import org.apache.lucene.index.NoMergePolicy;
   import org.apache.lucene.index.IndexWriterConfig;
   import org.apache.lucene.index.SegmentInfos;
   import org.apache.lucene.index.DirectoryReader;
   import org.apache.lucene.store.MMapDirectory;
   import org.apache.lucene.tests.util.LineFileDocs;
   
   // export LUCENE_VERSION=9.12.3; java -fullversion; ./gradlew jar; javac -verbose /home/mike/TestSILeak.java -cp lucene/core/build/libs/lucene-core-$LUCENE_VERSION-SNAPSHOT.jar:lucene/test-framework/build/libs/lucene-test-framework-$LUCENE_VERSION-SNAPSHOT.jar; java -cp /home/mike:lucene/core/build/libs/lucene-core-$LUCENE_VERSION-SNAPSHOT.jar:lucene/test-framework/build/libs/lucene-test-framework-$LUCENE_VERSION-SNAPSHOT.jar:lucene/codecs/build/libs/lucene-codecs-$LUCENE_VERSION-SNAPSHOT.jar:/home/mike/.gradle/wrapper/dists/gradle-8.14-bin/38aieal9i53h9rfe7vjup95b9/gradle-8.14/lib/junit-4.13.2.jar:/home/mike/.gradle/wrapper/dists/gradle-8.14-bin/38aieal9i53h9rfe7vjup95b9/gradle-8.14/lib/hamcrest-core-1.3.jar:/home/mike/.gradle/caches/modules-2/files-2.1/com.carrotsearch.randomizedtesting/randomizedtesting-runner/2.8.1/55ffe691e90d31ab916746516654b5701e532d6f/randomizedtesting-runner-2.8.1.jar TestSILeak indextmp
   
   public class TestSILeak {
   
     public static void checkMapFile() throws IOException {
       Path filePath = Path.of("/proc/" + ProcessHandle.current().pid() + "/maps");
       long totLineCount = Files.lines(filePath).count();
   
       long siLineCount = Files.lines(filePath)
         .filter(line -> line.contains(".si"))
         .count();
   
       System.out.println("MAP COUNT (from " + filePath + "): " + totLineCount + " [" + siLineCount + " si files]");
     }
   
     public static void main(String[] args) throws IOException, InterruptedException {
   
       if (args.length != 1) {
         System.out.println("Usage: java TestMSILeak <index dir>");
         System.exit(-1);
       }
       MMapDirectory dir = new MMapDirectory(Paths.get(args[0]));
       System.out.println("map file: /proc/" + ProcessHandle.current().pid() + "/maps");
   
       IndexWriterConfig iwc = new IndexWriterConfig(new StandardAnalyzer());
       iwc.setOpenMode(IndexWriterConfig.OpenMode.CREATE);
       iwc.setUseCompoundFile(false);
       // so we are sure we get the N flushed segments:
       iwc.setMergePolicy(NoMergePolicy.INSTANCE);
   
       Random r = new Random(42 * 17);
       LineFileDocs lfd = new LineFileDocs(r);
   
       try (IndexWriter writer = new IndexWriter(dir, iwc)) {
         for(int iter=0;iter<10;iter++) {
           for(int i=0;i<1000;i++) {
             writer.addDocument(lfd.nextDoc());
           }
           writer.commit();
         }
       }
   
       // hold a reader open
       DirectoryReader reader = DirectoryReader.open(dir);
       System.out.println("\nREADER: " + reader);
   
       while (true) {
         // each open will add the N _seg.si files as new maps, until we hit the OS limit:
         SegmentInfos sis = SegmentInfos.readLatestCommit(dir);
         //System.out.println("SIS: " + sis);
         checkMapFile();
         Thread.sleep(1000);
       }
     }
   }
   ```
   
   Check out 9.12.x branch and run that long command-line comment, tweaking the paths properly to your env, and it produces output something like this:
   
   ```
   openjdk full version "21.0.7+6"
   Starting a Gradle Daemon, 4 busy and 4 incompatible Daemons could not be reused, use --status for details
   BUILD SUCCESSFUL in 6s
   WARNING: A restricted method in java.lang.foreign.Linker has been called
   WARNING: java.lang.foreign.Linker::downcallHandle has been called by the unnamed module
   WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for this module
   
   Aug 14, 2025 8:05:57 AM org.apache.lucene.store.MemorySegmentIndexInputProvider <init>
   INFO: Using MemorySegmentIndexInput and native madvise support with Java 21 or later; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
   map file: /proc/206135/maps
   Aug 14, 2025 8:05:57 AM org.apache.lucene.internal.vectorization.VectorizationProvider lookup
   WARNING: Java vector incubator module is not readable. For optimal vector performance, pass '--add-modules jdk.incubator.vector' to enable Vector API.
   
   READER: StandardDirectoryReader(segments_z:106 _p(9.12.3):C1000:[diagnostics={os=Linux, java.vendor=Arch Linux, os.arch=amd64, os.version=6.15.2-arch1-1, lucene.version=9.12.3, source=flush, timestamp=1755173157857, java.runtime.version=21.0.7+6}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=1\
   1pns8d06il0choqhaknlj9aj _q(9.12.3):C1000:[diagnostics={os=Linux, java.vendor=Arch Linux, os.arch=amd64, os.version=6.15.2-arch1-1, lucene.version=9.12.3, source=flush, timestamp=1755173158149, java.runtime.version=21.0.7+6}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=11pns8d06il0choqhaknlj9\
   an _r(9.12.3):C1000:[diagnostics={os=Linux, java.vendor=Arch Linux, os.arch=amd64, os.version=6.15.2-arch1-1, lucene.version=9.12.3, source=flush, timestamp=1755173158408, java.runtime.version=21.0.7+6}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=11pns8d06il0choqhaknlj9ar _s(9.12.3):C1000:[d\
   iagnostics={os=Linux, java.vendor=Arch Linux, os.arch=amd64, os.version=6.15.2-arch1-1, lucene.version=9.12.3, source=flush, timestamp=1755173158693, java.runtime.version=21.0.7+6}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=11pns8d06il0choqhaknlj9av _t(9.12.3):C1000:[diagnostics={os=Linux, \
   java.vendor=Arch Linux, os.arch=amd64, os.version=6.15.2-arch1-1, lucene.version=9.12.3, source=flush, timestamp=1755173158985, java.runtime.version=21.0.7+6}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=11pns8d06il0choqhaknlj9az _u(9.12.3):C1000:[diagnostics={os=Linux, java.vendor=Arch Linux\
   , os.arch=amd64, os.version=6.15.2-arch1-1, lucene.version=9.12.3, source=flush, timestamp=1755173159271, java.runtime.version=21.0.7+6}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=11pns8d06il0choqhaknlj9b3 _v(9.12.3):C1000:[diagnostics={os=Linux, java.vendor=Arch Linux, os.arch=amd64, os.ve\
   rsion=6.15.2-arch1-1, lucene.version=9.12.3, source=flush, timestamp=1755173159574, java.runtime.version=21.0.7+6}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=11pns8d06il0choqhaknlj9b7 _w(9.12.3):C1000:[diagnostics={os=Linux, java.vendor=Arch Linux, os.arch=amd64, os.version=6.15.2-arch1-1, \
   lucene.version=9.12.3, source=flush, timestamp=1755173159887, java.runtime.version=21.0.7+6}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=11pns8d06il0choqhaknlj9bb _x(9.12.3):C1000:[diagnostics={os=Linux, java.vendor=Arch Linux, os.arch=amd64, os.version=6.15.2-arch1-1, lucene.version=9.12.3,\
    source=flush, timestamp=1755173160211, java.runtime.version=21.0.7+6}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=11pns8d06il0choqhaknlj9bf _y(9.12.3):C1000:[diagnostics={os=Linux, java.vendor=Arch Linux, os.arch=amd64, os.version=6.15.2-arch1-1, lucene.version=9.12.3, source=flush, timesta\
   mp=1755173160536, java.runtime.version=21.0.7+6}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=11pns8d06il0choqhaknlj9bj)
   MAP COUNT (from /proc/206135/maps): 512 [10 si files]
   MAP COUNT (from /proc/206135/maps): 521 [20 si files]
   MAP COUNT (from /proc/206135/maps): 531 [30 si files]
   MAP COUNT (from /proc/206135/maps): 541 [40 si files]
   MAP COUNT (from /proc/206135/maps): 551 [50 si files]
   MAP COUNT (from /proc/206135/maps): 561 [60 si files]
   MAP COUNT (from /proc/206135/maps): 571 [70 si files]
   MAP COUNT (from /proc/206135/maps): 581 [80 si files]
   MAP COUNT (from /proc/206135/maps): 591 [90 si files]
   MAP COUNT (from /proc/206135/maps): 601 [100 si files]
   MAP COUNT (from /proc/206135/maps): 611 [110 si files]
   MAP COUNT (from /proc/206135/maps): 621 [120 si files]
   MAP COUNT (from /proc/206135/maps): 631 [130 si files]
   MAP COUNT (from /proc/206135/maps): 641 [140 si files]
   MAP COUNT (from /proc/206135/maps): 651 [150 si files]
   MAP COUNT (from /proc/206135/maps): 661 [160 si files]
   MAP COUNT (from /proc/206135/maps): 671 [170 si files]
   MAP COUNT (from /proc/206135/maps): 681 [180 si files]
   MAP COUNT (from /proc/206135/maps): 691 [190 si files]
   MAP COUNT (from /proc/206135/maps): 701 [200 si files]
   MAP COUNT (from /proc/206135/maps): 711 [210 si files]
   MAP COUNT (from /proc/206135/maps): 721 [220 si files]
   MAP COUNT (from /proc/206135/maps): 731 [230 si files]
   MAP COUNT (from /proc/206135/maps): 741 [240 si files]
   MAP COUNT (from /proc/206135/maps): 751 [250 si files]
   MAP COUNT (from /proc/206135/maps): 761 [260 si files]
   MAP COUNT (from /proc/206135/maps): 771 [270 si files]
   MAP COUNT (from /proc/206135/maps): 781 [280 si files]
   MAP COUNT (from /proc/206135/maps): 791 [290 si files]
   MAP COUNT (from /proc/206135/maps): 801 [300 si files]
   MAP COUNT (from /proc/206135/maps): 811 [310 si files]
   MAP COUNT (from /proc/206135/maps): 821 [320 si files]
   MAP COUNT (from /proc/206135/maps): 831 [330 si files]
   MAP COUNT (from /proc/206135/maps): 841 [340 si files]
   MAP COUNT (from /proc/206135/maps): 851 [350 si files]
   MAP COUNT (from /proc/206135/maps): 861 [360 si files]
   MAP COUNT (from /proc/206135/maps): 871 [370 si files]
   MAP COUNT (from /proc/206135/maps): 881 [380 si files]
   MAP COUNT (from /proc/206135/maps): 891 [390 si files]
   MAP COUNT (from /proc/206135/maps): 901 [400 si files]
   ```
   
   I'm not sure why my gradle leaks daemons like a sieve.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3191347502

   > > Maybe setting `-Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits` to a smallish value (defaults to 1024) would work too, not sure.
   > 
   > I tested `-Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits=1` and it does indeed prevent the leak with my small test case on 9.12.x, JDK 21:
   > 
   > ```
   > MAP COUNT (from /proc/234634/maps): 508 [0 si files]
   > MAP COUNT (from /proc/234634/maps): 508 [0 si files]
   > MAP COUNT (from /proc/234634/maps): 508 [0 si files]
   > MAP COUNT (from /proc/234634/maps): 508 [0 si files]
   > MAP COUNT (from /proc/234634/maps): 508 [0 si files]
   > MAP COUNT (from /proc/234634/maps): 508 [0 si files]
   > MAP COUNT (from /proc/234634/maps): 508 [0 si files]
   > MAP COUNT (from /proc/234634/maps): 508 [0 si files]
   > ```
   
   I assume my non sysprop fix also fixes the issue:
   
   > As a quick workaround for 9.x one could use:
   > ```java
   > mmapDir.setGroupingFunction(fn -> fn.endsWith(".si") ? Optional.empty() : MMapDirectory.GROUP_BY_SEGMENT.apply(fn));
   > ```
   
   Uwe
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3189076886

   In 9.12.x I see the index input being opened with IOContext.READ, from `SegmentInfos.parseSegmentInfos`, which means that it will not use a confined arena. 
   
   ```
   	at org.apache.lucene.store.MemorySegmentIndexInputProvider.openInput(MemorySegmentIndexInputProvider.java:71)
   	at org.apache.lucene.store.MemorySegmentIndexInputProvider.openInput(MemorySegmentIndexInputProvider.java:33)
   	at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:394)
   	at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:156)
   	at org.apache.lucene.codecs.lucene99.Lucene99SegmentInfoFormat.read(Lucene99SegmentInfoFormat.java:94)
   	at org.apache.lucene.index.SegmentInfos.parseSegmentInfos(SegmentInfos.java:419)
   	at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:376)
   	at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:312)
   ...
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3190847167

   I'm in awe. I asked chatgpt to recreate what I have written in the past and it did a flawless job, actually. Wow. 
   
   Anyway. https://github.com/apache/lucene/pull/15077 has what I had in mind - I wonder if it'll be able to track the problem.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3190805426

   Amazingly, ChatGPT also got the gist of it. Wild. https://chatgpt.com/c/689edb07-79c8-8326-bfe2-6e9b28f5a452
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3211113499

   Hi,
   
   I think this is a real bug. The problem is that the close code has this while spin-loop (because it may happen that Arena's close may not work on highly concurrent workloads). The problem is that the spin-loop checks the inner scope's isAlive(), whis stays alive if the refcount does not reach "0". So basically the RefCountedSharedArena code also needs to keep track of the inner scope!
   
   I wonder why it works correct at all! @ChrisHegarty ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3191348389

   > I assume my non sysprop fix also fixes the issue:
   
   +1, I'll test.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3189092240

   Maybe this explains also why so many people have problems with 9.x, especially when they reopen indexes all the time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3190782118

   > > > I'm not sure why my gradle leaks daemons like a sieve.
   > > 
   > > 
   > > Hahaha. 🤓🐰
   > 
   > This is probably because you're using different vms or gradle options (jvm options). I can't say. You (we) have three options - run without the daemon (Robert's choice), lower daemon expiration timeout, write our own build system from scratch. I'm recently leaning towards (3)...
   
   Let's start a project called "Policeman Build Slave" which is named non-inclusively because your computer has to do what you want and not start any arbitrary daemons. Policeman Build Slave would be written in pure Java with its own bytecode-compiled task language and forbids any bullshit by default with integrated linters everywhere.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3190862992

   Oh, sorry. I checked this on main, not 9x. Maybe it'll help to diagnose the issue on 9x though!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3190971672

   > The tests seem to pass with the ref counting trick for me. Maybe Mike's repro will fail though - something worth trying.
   > Oh, sorry. I checked this on main, not 9x. Maybe it'll help to diagnose the issue on 9x though!
   
   Actually the reason for the above issue is already known in 9.x, it is because the .si files are not opened with READONCE.
   
   Helping to diagnose leaking shared arenas may help people in #15054, although I don't think the OpenSearch issue is really related to a bug in those reference counting, it is just that they have too many mappings from the beginning and when using MMapDircetory for replication, too (which I don't uderstand why they do this), its a problem in OS and not Lucene. The Lucene directory abstractions are optimized to read/write indexes (random access) but mmap is not made for sequential access to copy data over the wire. You can do this with READONCE but then you have to live with the fact that it needs to be single-threaded.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3191225447

   The reason I didn't add unit tests for RefCountedSharedArena was that it was in the versioned section when first added. We should add unit tests for it now, at least in main.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "ayinresh (via GitHub)" <gi...@apache.org>.
ayinresh commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3210953067

   Heads up that there may be other issues with `RefCountedSharedArena`. On 10.2.2, we hit an infinite loop at the following stacktrace:
   ```
   java.lang.Thread.State: RUNNABLE
     at org.apache.lucene.store.RefCountedSharedArena.close(RefCountedSharedArena.java:123)
     at org.apache.lucene.store.MemorySegmentIndexInput.close(MemorySegmentIndexInput.java:702)
     at org.apache.lucene.util.IOUtils.close(IOUtils.java:85)
     at org.apache.lucene.util.IOUtils.close(IOUtils.java:72)
     at org.apache.lucene.codecs.lucene90.Lucene90CompoundReader.close(Lucene90CompoundReader.java:150)
     at org.apache.lucene.util.IOUtils.close(IOUtils.java:85)
     at org.apache.lucene.util.IOUtils.close(IOUtils.java:72)
     at org.apache.lucene.index.SegmentCoreReaders.decRef(SegmentCoreReaders.java:172)
     at org.apache.lucene.index.SegmentReader.doClose(SegmentReader.java:223)
     at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:258)
     at org.apache.lucene.index.StandardDirectoryReader$$Lambda/0x0000000080e0cd48.accept(Unknown Source)
     at org.apache.lucene.util.IOUtils.lambda$applyToAll$0(IOUtils.java:467)
     at org.apache.lucene.util.IOUtils$$Lambda/0x0000000080e03c20.close(Unknown Source)
     at org.apache.lucene.util.IOUtils.close(IOUtils.java:85)
     at org.apache.lucene.util.IOUtils.applyToAll(IOUtils.java:466)
     at org.apache.lucene.index.StandardDirectoryReader.doClose(StandardDirectoryReader.java:482)
     at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:258)
     at org.apache.lucene.search.SearcherManager.decRef(SearcherManager.java:136)
     at org.apache.lucene.search.SearcherManager.decRef(SearcherManager.java:52)
     at org.apache.lucene.search.ReferenceManager.release(ReferenceManager.java:264)
     at org.apache.lucene.search.ReferenceManager.swapReference(ReferenceManager.java:57)
     - locked <0x00000403f312ef18> (a org.apache.lucene.search.SearcherManager)
     at org.apache.lucene.search.ReferenceManager.close(ReferenceManager.java:139)
     - locked <0x00000403f312ef18> (a org.apache.lucene.search.SearcherManager)
   ```
   It appears that `MemorySegmentIndexInput` repeatedly tries to close the `RefCountedSharedArena` [here](https://github.com/apache/lucene/blob/32e97a6f66157a98f35ba98d983ad0d5eb5de4b9/lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java#L718), but `RefCountedSharedArena` always fails [here](https://github.com/apache/lucene/blob/b1c1770e8052a74fd30e3520e4fd662e2724bd1c/lucene/core/src/java/org/apache/lucene/store/RefCountedSharedArena.java#L107).
   
   The effect is that closing the searcher blocks indefinitely. We discovered this as a root cause of a production outage this morning, so I haven't had time to read up fully on this code / thread, and apologies for not providing more context here. Planning on doing a more detailed writeup and filing as a separate issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "ayinresh (via GitHub)" <gi...@apache.org>.
ayinresh commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3211517180

   Hi @uschindler,
   
   Thank you for opening the issue!
   
   Yes this is custom usage of Lucene (not opensearch/ES). We do call [ReferenceManager::close](https://github.com/apache/lucene/blob/b1c1770e8052a74fd30e3520e4fd662e2724bd1c/lucene/core/src/java/org/apache/lucene/search/ReferenceManager.java#L134) while it is in use by other threads, but the API indicates that this is safe:
   
   ```
   /**
      * Closes this ReferenceManager to prevent future {@link #acquire() acquiring}. A reference
      * manager should be closed if the reference to the managed resource should be disposed or the
      * application using the {@link ReferenceManager} is shutting down. The managed resource might not
      * be released immediately, if the {@link ReferenceManager} user is holding on to a previously
      * {@link #acquire() acquired} reference. The resource will be released once when the last
      * reference is {@link #release(Object) released}. Those references can still be used as if the
      * manager was still active.
      *
      * <p>Applications should not {@link #acquire() acquire} new references from this manager once
      * this method has been called. {@link #acquire() Acquiring} a resource on a closed {@link
      * ReferenceManager} will throw an {@link AlreadyClosedException}.
      *
      * @throws IOException if the underlying reader of the current reference could not be closed
      */
   ```
   
   Based on that documentation we do the following (which we believed was safe):
   - A single background thread indexes documents and periodically creates a new NRT `SearcherManager`
   - Query threads acquire the `SearcherManager` and can hold it for extended periods of time
   - When the background thread decides that the `SearcherManager` should expire, it calls `SearcherManager::close`. The expectation is that the last query thread to release the `SearcherManager` will actually be the one to execute [doClose](https://github.com/apache/lucene/blob/cb16075c5fd5c4bbcb0fab561e185fe2e3f8e939/lucene/core/src/java/org/apache/lucene/index/IndexReader.java#L224)
   - [It might seem strange that we don't just use `maybeRefresh`, but we implement point in time search and need to keep multiple searchers around for long periods of time]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "schmika (via GitHub)" <gi...@apache.org>.
schmika commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3655868198

   Hi,
   we are running Apache Solr 9.10.0 and Lucene 9.12.3 and also encountered problems due to large numbers of memory mapped segments. In our case, #15078 did not help, because `MMapDirectory` seems to use its own default value of 1024 for `sharedArenaMaxPermits`: https://github.com/apache/lucene/blob/f965e930673c1f5cb478dc6a8907f5fc4ef7b539/lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java#L504-L505
   Explicitly setting `-Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits=64` prevented the issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3211140733

   The code in MemorySegmentIndexInput assumes that the call of close can be repeated if it is failing with IllegalStateException, but the RefCountedSharedArena may fail to handle this correctly (what happens op exception?).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3211944928

   Hi @ayinresh,
   this should be the case, but I am not 100% sure if it still might happen that the close is rejected by the JVM after the ReferenceManager has decided to close because maybe the other thread may still not have enetered a safepoint to make sure it is really done with accessing the already closed IndexInput? I am not 100% sure about this.
   
   I will try to create a reproducer what I figured out in the opened issue (we have a test for this that I can beef up to use a refcounted arena). Basically I'd like to remove the extra check in the RefCountedArena, plain simple. There is no need to throw an IllegalStateException pro-actively and just hand over the closing to the undelying `Arena#close()`.
   
   In addition are you really sure that no other thread that maybe still getting stored fields after all other threads have released the searcher? RefrenceManager only ensures no _new_ threads can aquire a searcher, but nothing prevents other threads from using an open searcher that might have been released!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3190860143

   The tests seem to pass with the ref counting trick for me. Maybe Mike's repro will fail though - something worth trying.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3191334855

   > Amazingly, ChatGPT also got the gist of it. Wild. https://chatgpt.com/c/689edb07-79c8-8326-bfe2-6e9b28f5a452
   
   Curiously ChatGPT won't load this conversation ... just a red error box saying "Unable to load conversation....".  Are you sure you published / made it public?  Sharing is caring.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3191379187

   Wow, thank you everyone for the fast ideas, digging, and fixes!  I wasn't even sure I should bring this issue up (I thought maybe it was more Amazon's crazy heavy Lucene usage), but I'm really happy I did!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3191355454

   > > I assume my non sysprop fix also fixes the issue:
   > 
   > +1, I'll test.
   
   Yup, it stops the leak cold :)  Doesn't let an `_N.si` files into the arena!
   
   ```
   MAP COUNT (from /proc/236326/maps): 462 [0 si files]
   MAP COUNT (from /proc/236326/maps): 462 [0 si files]
   MAP COUNT (from /proc/236326/maps): 462 [0 si files]
   MAP COUNT (from /proc/236326/maps): 462 [0 si files]
   MAP COUNT (from /proc/236326/maps): 462 [0 si files]
   MAP COUNT (from /proc/236326/maps): 462 [0 si files]
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3196239140

   > > Curiously ChatGPT won't load this conversation ... just a red error box saying "Unable to load conversation....". Are you sure you published / made it public? Sharing is caring.
   > 
   > It should be public, yes. Strange. This is a follow-up - I was curious what it could come up with but it's not any better in the end. https://chatgpt.com/share/689edbeb-b298-8011-b031-d2dbcc882e3c
   
   Now it loads fine for me -- very cool interaction and simple prompting!  Amazing ChatGPT can understand, and then (sort of) simplify the code. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "ayinresh (via GitHub)" <gi...@apache.org>.
ayinresh commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3222001450

   @uschindler
   
   After investigating the production outage, you are absolutely correct that we were illegally using a searcher that had already been released. I found an error log with `AlreadyClosedException` thrown from an async thread just before the close thread enters the infinite loop. I will fix this usage.
   
   It seems we did get rather unlucky here - rather than experiencing a runtime exception on one thread, we entered into a fatal state (infinite loop) which brought down our node (the spinning thread held a critical lock...).
   
   Thanks for fixing this bug! I see https://github.com/apache/lucene/pull/15112 already merged.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] MMapDirectory sometimes "leaks" 1000s of maps [lucene]

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #15068:
URL: https://github.com/apache/lucene/issues/15068#issuecomment-3223066748

   Hi,
   
   > After investigating the production outage, you are absolutely correct that we were illegally using a searcher that had already been released. I found an error log with `AlreadyClosedException` thrown from an async thread just before the close thread enters the infinite loop. I will fix this usage.
   > 
   > It seems we did get rather unlucky here - rather than experiencing a runtime exception on one thread, we entered into a fatal state (infinite loop) which brought down our node (the spinning thread held a critical lock...).
   > 
   > Thanks for fixing this bug! I see [#15112](https://github.com/apache/lucene/pull/15112) already merged.
   
   Thanks for the confirmation!
   
   Regarding the severity of this bug see the linked PR- I gave a comment. The fix will be in Lucene 10.3 and Lucene 11 (main). If and only if we release a 9.12.x version to fix bugs, we should really backport it there, too. But using setGrouping(NO_GROUPING) shoudl be as a workaround if you hit this, to allow better debugging. But the underlying problem is generally in user's code.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org