You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "drabe8 (via GitHub)" <gi...@apache.org> on 2025/08/24 22:37:15 UTC
[I] Deadlock between IndexWriter, IndexWriterConfig, ConcurrentMergeScheduler [lucene]
drabe8 opened a new issue, #15119:
URL: https://github.com/apache/lucene/issues/15119
### Description
Similar to https://github.com/apache/lucene/issues/12419
In our use case, we create many IndexWriters on the fly, dependent on the workload assigned to our process. (We use a ByteBuffersDirectory to create discrete indexes for a small subset of the data that a given user is working with.)
We see one thread constructing an IndexWriterConfig, waiting for the IndexWriter Class Init monitor:
```
java.lang.Thread.State: RUNNABLE
at java.lang.Class.forName0(java.base@21.0.7.0.101/Native Method)
- waiting on the Class initialization monitor for org.apache.lucene.index.IndexWriter
at java.lang.Class.forName(java.base@21.0.7.0.101/Class.java:421)
at java.lang.Class.forName(java.base@21.0.7.0.101/Class.java:412)
at org.apache.lucene.internal.tests.TestSecrets.lambda$static$0(TestSecrets.java:43)
at org.apache.lucene.internal.tests.TestSecrets$$Lambda/0x00007ff82101b458.accept(Unknown Source)
at org.apache.lucene.internal.tests.TestSecrets.<clinit>(TestSecrets.java:51)
at org.apache.lucene.index.ConcurrentMergeScheduler.<clinit>(ConcurrentMergeScheduler.java:939)
at org.apache.lucene.index.LiveIndexWriterConfig.<init>(LiveIndexWriterConfig.java:130)
at org.apache.lucene.index.IndexWriterConfig.<init>(IndexWriterConfig.java:145)
```
And another thread constructing an IndexWriter, waiting for the TestSecrets Class Init monitor:
```
java.lang.Thread.State: RUNNABLE
at org.apache.lucene.index.IndexWriter.<clinit>(IndexWriter.java:6597)
- waiting on the Class initialization monitor for org.apache.lucene.internal.tests.TestSecrets
```
And thousands of threads are stuck trying to create an IndexWriterConfig, stuck waiting for the ConcurrentMergeScheduler Class Init monitor:
```
java.lang.Thread.State: RUNNABLE
at org.apache.lucene.index.LiveIndexWriterConfig.<init>(LiveIndexWriterConfig.java:130)
- waiting on the Class initialization monitor for org.apache.lucene.index.ConcurrentMergeScheduler
at org.apache.lucene.index.IndexWriterConfig.<init>(IndexWriterConfig.java:145)
```
I believe this is a legitimate use case; when the deadlock occurs, it is a major problem for us.
### Version and environment details
Lucene 10.2.2
Java 21.0.7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Deadlock between IndexWriter, IndexWriterConfig, ConcurrentMergeScheduler [lucene]
Posted by "danmuzi (via GitHub)" <gi...@apache.org>.
danmuzi commented on issue #15119:
URL: https://github.com/apache/lucene/issues/15119#issuecomment-4119725823
@drabe8
Will be fixed in Lucene 10.5
PR : https://github.com/apache/lucene/pull/15864
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Deadlock between IndexWriter, IndexWriterConfig, ConcurrentMergeScheduler [lucene]
Posted by "danmuzi (via GitHub)" <gi...@apache.org>.
danmuzi closed issue #15119: Deadlock between IndexWriter, IndexWriterConfig, ConcurrentMergeScheduler
URL: https://github.com/apache/lucene/issues/15119
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Deadlock between IndexWriter, IndexWriterConfig, ConcurrentMergeScheduler [lucene]
Posted by "ayinresh (via GitHub)" <gi...@apache.org>.
ayinresh commented on issue #15119:
URL: https://github.com/apache/lucene/issues/15119#issuecomment-3234589059
@drabe8
Not an expert, but I believe this is a type of class loading deadlock, which has occurred numerous times in the past (e.g. [LUCENE-6482](https://issues.apache.org/jira/browse/LUCENE-6482)).
We just hit this today as well (Lucene `10.2.2`, Java `21.0.8`). It occurred just after node restart (when class loading would occur for the first time). Our thread dump showed over 1K `RUNNABLE` threads with `in Object.wait()`, and `waiting on the Class initialization monitor for <class>` at the top of the stacktrace. Specifically, threads are blocked on class loading for the following classes:
- `org.apache.lucene.codecs.lucene90.blocktree.SegmentTermsEnum` (1077 threads)
- `org.apache.lucene.search.TermInSetQuery$SetEnum` (51 threads)
- `org.apache.lucene.index.BaseTermsEnum` (3 threads)
- `org.apache.lucene.index.SortedDocValuesTermsEnum` (17 threads)
- `org.apache.lucene.index.TermsEnum` (1 thread)
After some basic research, it seems the easiest fix is to try to preload these classes at startup using `java.lang.Class::forName` before any other threads attempt to utilize Lucene and trigger class loading concurrently (risking deadlock). I'm not sure which classes need to be preloaded, but I would start with the classes present in our thread dumps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org