You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "ChrisHegarty (via GitHub)" <gi...@apache.org> on 2024/06/10 12:33:56 UTC
[I] Examine performance of individual data accessor methods of MemorySegmentIndexInput [lucene]
ChrisHegarty opened a new issue, #13325:
URL: https://github.com/apache/lucene/issues/13325
Examine performance of individual data accessor methods of MemorySegmentIndexInput, as compared to the ByteBufferIndexInput. Specifically getByte, getShort, and getBytes.
This issue has been filed to raise awareness of a seemingly perf degradation of MemorySegmentIndexInput when used heavily in decompression, see https://github.com/dacapobench/dacapobench/issues/264. I've not yet investigated the low-level segment guards, but maybe there is something here (or in the VarHandles) that is not optimising?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput [lucene]
Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2083195761
Thanks for your quick reply @uschindler. Your analysis makes sense to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput when IndexInputs are closed in other threads (deoptimizations,...) [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2217896710
> > Possibly group multiple files into one arena. One Idea that came into my mind: MMapDirectory groups files belonging to the same segment together and uses a single Arena for them. Behind the scenes it keeps a refcount for each group of files. So whenever a file is opened the refcount for that group increases and when it is closed it decreases. When the refcount goes to 0 the Arena is closed.
>
> This seems really promising as well. I may try to pick this up unless anyone else is already looking into it? This seems like the right thing to do semantically even, in addition to the potential performance benefits.
I had no time to look into this. It should be doable with a instance ConcurrentHashMap, not sure where to place this instance (as it is also using those providers). So the problematic part is where to place this (as it cannot be in MMapDirectory where it should belong as it uses Java 21 preview apis).
You can give it a try, but i have the feeling it gets tricky on the actual implementation. We may need to add some abstraction into MMapDirectory instances (the ConcurrentHashMap) with the refcounter and a pointer to some "hidden" payload (the areana) which is picked by the private MemorySegmentIndexInputProvider.
We can work together. If you have a better idea, tell me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput when IndexInputs are closed in other threads (deoptimizations,...) [lucene]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2242901661
I filed an issue in Solr: https://issues.apache.org/jira/browse/SOLR-17375
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput when IndexInputs are closed in other threads (deoptimizations,...) [lucene]
Posted by "magibney (via GitHub)" <gi...@apache.org>.
magibney commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2218097364
I extended the `Arena` interface to track refCount for https://github.com/apache/lucene/pull/13555 -- curious what you think of this approach?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput when IndexInputs are closed in other threads (deoptimizations,...) [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2238591954
Hi David,
sorry for late response. In fact there problem is quite simple. Every IndexInput#close() costs time in the thread calling this. This time is linear to the number of threads (in plain Java 19-21 at moment).
With the fixes by JDK people it will change a bit:
- It will still be somehow linear to the total number of threads, but gets faster when not all threads are actually inside a MemorySement read
- The will be no overhead anymore on consuming threads that don't use MemorySegments; at the moment it deoptimizes all threads.
So with the JDK changes to be applied to Java 22+ (and maybe also 21) the overhead will be almost zero for threads querying Lucene. The work for closing only minimally changes (if at all).
So the best we can do is: Don't concurently close IndexInputs in the main threads because on heavy loaded Solr clusters this can take a lot of time. So the idea is to do all "maintenance" on Lucene IndexSearchers/IndexReaders in a small & separate pool of threads. Actively reopen a DirectoryReader (after commit or softcommit) can be done in main thread, but closing the old one can easily be deferred to a small maintenance thread pool.
There is no way around this, as the ByteBuffer based MMap will no lobger be possible with later Java versions, as sun.misc.Unsafe will be decommissioned in Java 24 latest. So the new code and its additional safety we have to live with. And that's good. My suggestion would be to to also rethink Solr's thread usage. 10.000 threads is incredibly wrong! Possibly use virtual threads. If you do this, the problem will also go away.
Uwe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput when IndexInputs are closed in other threads (deoptimizations,...) [lucene]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2231607373
Solr already takes care to ensure a SolrCore is closed while nobody is accessing it via its refcount mechanism on the SolrCore. I understand the JVM doesn't know that, and needs to take precautions. That it's O(threads) means the Solr project should prioritize taming its 10k-bounded Jetty thread pool for this new reason.
> basically reopening IndexReaders (closing old segments, open new segments). Basically a Solr softcommit, but also applies to commits
Woah; per commit! I thought you just meant the equivalent of opening and closing a SolrCore, which has an index -- somewhat rare. Are you saying that, ideally, Solr should close the _previous_ IndexReader (tied to a Searcher that will close) asynchronously? And you suggested by one thread... does it need to be one thread for the whole JVM (this worries me) or one per core?
> the JDK maintainers recommend to possibly do the closing of indexes asynchronously in production environments
It's okay if closing a SolrCore is a bit slower. In practice, this means a core UNLOAD request would take longer thus deleting a collection takes longer (albeit is already parallelized per replica). Shutting down Solr would take longer; okay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput [lucene]
Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2084934317
Thanks for jumping on this so quickly @uschindler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2083061482
The issue is analyzed already: The benchmark is wrong. It closes and opens indexes all the time. Due to the additional safety applied on closing indexes (to not crush the JVM with SIGSEGV), the JVM signals to all other threads that they need to go into safepoint to do the thread-local handshake.
The benchmakr should be rewritten to use a single IndexReader instance with many threads and not open a separate one for short time all the time in search threads. This causes thread-local-handshake havoc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2085320446
They identified that deoptimizations are indeed the cause of all this. When you trigger a thread-local handshake by closing an IndexInput, all code in other threads which runs at the same time and accessing memory segments gets deoptimized somehow. I have no idea how it enters optimized mode again, but this seems to cause the slowdown for that benchamrk.
I am not sure if we can open issues about this in OpenJDK to optimize this. Why does the code need to be deoptimized when we enter a safepoint/handshake? It just traps with a segv and enters safepoint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler closed issue #13325: Examine performance of individual data accessor methods of MemorySegmentIndexInput
URL: https://github.com/apache/lucene/issues/13325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput when IndexInputs are closed in other threads (deoptimizations,...) [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2240961607
> Thanks for your detailed response!
>
> > The will be no overhead anymore on unrelated threads that don't use MemorySegments; at the moment it deoptimizes all threads.
>
> Much of the fundamental tasks of a search engine access the index and those will use the MemorySegment API. Won't the "deoptimize" part still be a problem?
No, see benchmarks. In fact only the top frame is deoptimized, but once the processing can go on the top frame is replaced by the orptimized code again. This is a typical VM operation that happens under other circumstances, too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput when IndexInputs are closed in other threads (deoptimizations,...) [lucene]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2240911730
Thanks for your detailed response!
> The will be no overhead anymore on unrelated threads that don't use MemorySegments; at the moment it deoptimizes all threads.
Much of the fundamental tasks of a search engine access the index and those will use the MemorySegment API. Won't the "deoptimize" part still be a problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput when IndexInputs are closed in other threads (deoptimizations,...) [lucene]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2241852198
I set a breakpoint in a debugger on IndexInput.close while running a Solr test and it's called in **lots** of places separately from closing an IndexReader. Lucene FieldsIndexWriter.finish *reads* a file it just finished writing, for example. If the underlying problem is more closely associated to the IndexInput and not the IndexReader, shouldn't IndexInput.close be internally async, and thus be a Lucene level change?
I also set a breakpoint on IndexReader.close. Lucene's SpellChecker.swapSearcher is invoked when the searcher changes (e.g. from a commit) via Solr's SpellCheckerListener. Hopefully the only effect leaving this is commits being slower.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2083005280
They say that it only happens on high concurrency. In addition: Does the benchmark use stored fields at all for measuring query performance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2084925574
We have the confimation: https://github.com/dacapobench/dacapobench/issues/264#issuecomment-2084907716
I think we can close this issue.
Anyways we should talk to the Hotspot people to figure out how we can improve the safepoint/thread-local handshake in a way that it does not get so expensive for other threads. To me it looks like on every close, all methods accessing MemorySegments get deoptimized, but I haven't looked into this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2083760951
> Thanks for your quick reply @uschindler. Your analysis makes sense to me.
If they don't respond, I will rewrite their benchmark to only open the index once and then hammer it with different threads. If this is then identical speed the issue seen is confirmed to be caused by the thread-local handshakes.
Although this is incorrect use of Lucene, we should still try to investigate with Hotspot people why the handshakes/safepoints always lead to some kind of deoptimization in all threads accessing MemorySegments. This is not necessarily bad in normal workloads, but could lead to slowdowns in NRT use cases with high index refresh frequencies.
We could also add an option to MMapDirectory so you can make it ready for purely single-threaded use. I am not sure if this is useful in practise, but we could easily allow this. If another thread then reads from an index file it will trigger IllegalStateException.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput when IndexInputs are closed in other threads (deoptimizations,...) [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2242171172
> I set a breakpoint in a debugger on IndexInput.close while running a Solr test and it's called in **lots** of places separately from closing an IndexReader. Lucene FieldsIndexWriter.finish _reads_ a file it just finished writing, for example.
This ChecksumIndexInput should be opened with READONCE, so it is confined. So there's no cost on closing:
https://github.com/apache/lucene/blob/cc3b412183c7acbb4a6c498e9babffba4e7d1f28/lucene/core/src/java/org/apache/lucene/codecs/lucene90/compressing/FieldsIndexWriter.java#L123
Uwe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput when IndexInputs are closed in other threads (deoptimizations,...) [lucene]
Posted by "magibney (via GitHub)" <gi...@apache.org>.
magibney commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2217862298
> Possibly group multiple files into one arena. One Idea that came into my mind: MMapDirectory groups files belonging to the same segment together and uses a single Arena for them. Behind the scenes it keeps a refcount for each group of files. So whenever a file is opened the refcount for that group increases and when it is closed it decreases. When the refcount goes to 0 the Arena is closed.
This seems really promising as well. I may try to pick this up unless anyone else is already looking into it? This seems like the right thing to do semantically even, in addition to the potential performance benefits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput when IndexInputs are closed in other threads (deoptimizations,...) [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2228936957
Yeah: https://github.com/openjdk/jdk/pull/20158#issuecomment-2228916752
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput when IndexInputs are closed in other threads (deoptimizations,...) [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2242174153
All ChecksumIndexInput are automatically opened using READONCE in main branch as those cannot get consumed by multiple threads. In the backport it's manually done; if we missed one place please open issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput when IndexInputs are closed in other threads (deoptimizations,...) [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2228749316
We also have another PR doing the same than @magibney's: https://github.com/apache/lucene/pull/13570
We also got some updates on the JDK front. Next to the fixes here (use confined segments for READONCE and grouping shared arenas) there are a few suggestions:
I have the feeling that Elasticsearch does not see the issue like Solr does under high load because the concurrency is different. @ChrisHegarty Is it true that Elasticsearch refreshes indexes in a sepate thread, so all close() calls are from a few threads only? In large Solr the installations the problem is mainly that every core looks like having its own threads that refresh indexes. Maybe it would be a good idea in Apache Solr to somehow have a maintenance thread which is responsible for refreshing indexes. The original patch by @magibney already showed that a static singleton lock around `Arena#close` looks like it helps. This in fact makes all closes sequential.
@dsmiley Maybe that's another solution to prevent this issue from happening: Introduce a global "maintenance task" in Solr which is responsible to reopen indexes. We could include that also in the SearcherManager tool? (cc @mikemccand).
I think with all those fixes here we have fixed most of the overhead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput when IndexInputs are closed in other threads (deoptimizations,...) [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2230338249
To be clear: With raising number of threads, the process of closing an indexinput gets slower. With recent Lucene improve,ments this is mitigated a bit by grouping files together and read READONCE files with a confined arena, but for all other IndexInputs which are in multithreaded use, the closing thread has to wait for the handshake on ALL other threads, so it does not scale well when there are many threads. This is needed for safety to ensure no other thread is accesssing the index while we are closing. There is no way to work around that as it is the required safety applied by the JVM (that we are currently missing in Lucene, this is why you can crush the whole VM when you close an IndexReader while threads are accessing it).
Because of this, the JDK maintainers recommend to possibly do the closing of indexes asynchronously in production environments. So basically in the normal processing of Solr, reopen the index as usual, but when actually closing the old IndexReader defer this to a single-threaded helper. This might also be a good idea for Elasticsearch.
This would allow to open the fresh indexes as soon as possible, but the old segments get closed in a delayed way. We can't implement this inside Lucene as we have no access to thread pools, it is up to the user to possibly close indexes async.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput when IndexInputs are closed in other threads (deoptimizations,...) [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2230139147
Basically the idea is to close indexes asynchronous in a single thread doing this to not block main threads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2082853451
I've also opened https://github.com/mikemccand/luceneutil/issues/267 to understand why our nightly benchmarks didn't notice this. @uschindler maybe you have an idea!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2158241513
Hi,
I reopened this to make a further investigation.
@dsmiley talked to me at Berlinbuzzwords and he also commented on #13146. He has seen a major slowdown on Apache Solr with Java 21 and using Lucene 9.8 (or like that). The issue there was partly 3x slower queries (as far as I understood) in an environment where many many Solr Cores with man many softcommits (IndexReader repoens for NRT) are happening at the same time. So think of about 1000 indexes on the same JVM and all of them are repopen indexes every second.
There is seems to be the problem that closing one IndexInput seems to deoptimize every other MemorySegment although not related / belonging to the same shared Arena.
I try to write a JMH benchmark without Lucene code to reproduce the issue and measure how large the issue is (depending on how often you close an IndexInput and how many parallel threads are using other MemorySegments, also those not tied to same Indexinput).
My idea is:
- Have many threads working on "hot" MemorySegments allocated with different shared Arenas.
- Have a few threads closing those Arenas.
Theoretically, the benchmark should not slow if the few threads close Arenas which are not bound to and hot workers.
What we have seen in dacapo bench is that it seems to affect all MemorySegments negatively, not only those which are affected by the close. of Arena.
Once I have the benchmark I will open issue at OpenJDK.
In the meantime, we could "preserve" the old Lucene 9.1 (the version without MemorySegments) in the "misc" module of Lucene as LegacyMMapDirectory. This would allow people to use it, but it is strinly discouraged. It won't work anymore with Java 24 (or around that time).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2083062574
So this issue is not in Lucene. The slowdown comes from incorrect use of Lucene. I am waiting for the confirmation by them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput when IndexInputs are closed in other threads (deoptimizations,...) [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2238603586
P.S.: The code optimizations were committed for JDK 23 already. Backport is under discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput when IndexInputs are closed in other threads (deoptimizations,...) [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2230135534
It is not a JDK bug, it is expected behaviour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput when IndexInputs are closed in other threads (deoptimizations,...) [lucene]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on issue #13325:
URL: https://github.com/apache/lucene/issues/13325#issuecomment-2229550743
What does "refreshing indexes" mean exactly? Solr does not have special threads for opening/closing indexes; they will typically happen on the common Jetty request thread pool. If the suggestion is _intended_ to ensure that only one thread open/close an index, I don't see why that would be a "maintenance task" but I do see it could easily be done simply by ensuring only one thread does this work on-demand. But I'd be concerned indexes would open slowly if you have many. Any way; feels like a work-around to the JDK bug and not something desirable in its own right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org