You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by David Smiley <ds...@apache.org> on 2024/08/21 17:23:19 UTC
FYI for those trying Java 21
For those trying Java 21 with Solr, if you find the performance to be
bad then try setting
-Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
This made a huge difference for me on our Solr nodes at work.
I filed the following ticket to Solr that ought to avoid the JDK
performance regression if done:
https://issues.apache.org/jira/browse/SOLR-17375
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
Re: FYI for those trying Java 21
Posted by Uwe Schindler via dev <de...@solr.apache.org>.
What I misses to add: Files opened with READ_ONCE are the way to go
anyways, because READ_ONCE files are designed to be kept open for short
time and only read with the same thread opening them (like the pattern
here). This reduces Java safepoints and deoptimization dramatically
because the arenas used are not "shared" and need no thread
synchronization and safepoints on closing.
So my recommenation is: In all Lucene versions user READ_ONCE for all
those shortly open files polled on many times to reduce contention and
JVM safepoints. In addition in Lucene 9 reduce the sysprop
org.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits to very low
values, but can also be tuned for Lucene 10 if you have many indexes on
same machine. Lower values increase cost of closing files but allow more
indexes in parallel.
Uwe
Am 20.02.2026 um 12:41 schrieb Uwe Schindler via dev:
> Hi,
>
> yes this looks a big problem. The functionality and heuristics for
> grouping arenas in MMapDirectory ist made for the usual index file
> names. Those filenames do not conform to the standard pattern of
> Lucene and therefore cannot be grouped by segment number and it also
> is not marked as metadata file. Basically what happens is that files
> in Lucene are opened once and to spare system resources and additional
> overhead while closing arenas is to group those together. The
> inappropiate named files get a new group on each open and they are not
> released in time and consume resources. MMapDir cant handle the files
> correctly.
>
> The problem with Lucene 9 is that it reserves slots for 1024 groups by
> default per index and reopening the same file over and over keeps
> those groups for much longer than expected. This can be mitigated by
> another system property (see MMapDirectory javadocs) to limit the
> number of groups to 64 or even lower (this was done in Lucene 10). The
> limit 1024 was way too large as every index can have up to 1024 arenas
> open at same time. The number of mapping allowed by the kernel (unless
> tuned) is only 65536 arenas / maps. So once you have 64 indexes and
> reopen the file over and over while keeping previous mmaps, you run
> out of mappings in the kernel.
>
> if you see this error for other files this is just a side effect of
> the open same file in a new group, at some point you run out of kernel
> resources after 64 indexes.
>
> Quick workaround:
> https://lucene.apache.org/core/9_12_0/core/org/apache/lucene/store/MMapDirectory.html#NO_GROUPING
> This restores behaviour of around Lucene 9.10 where all files are
> closed and all their mappings not cached.
>
> Or set following sysprop (defaults 1024) to much lower values (I'd
> recommend 16 if you have many indexes on same instance):
> org.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits
>
> (with the expense of slowdown on closing IndexInputs).
>
> Uwe
>
> Am 19.02.2026 um 18:21 schrieb Matthew Biscocho (BLOOMBERG/ 120 PARK):
>> Thanks Uwe for the info.
>>
>> We ran into this error in a bunch of different ways from Solr. I did
>> some digging and we ran into this with other places that should
>> probably have this change as well for example the
>> replication.properties file:
>>
>> https://github.com/apache/solr/blob/a36c3b511dbd56e929d93f2b2ed7cf94adb436e9/solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java#L1189
>>
>>
>>
>> Ran into it here that also include actual segments that probably
>> should be READONCE probably:
>> https://github.com/apache/solr/blob/a36c3b511dbd56e929d93f2b2ed7cf94adb436e9/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java#L1367
>>
>> But this issue still occurred with segment files that did do a READONCE
>>
>> java.io.IOException: Map failed:
>> MemorySegmentIndexInput(path=\"/var/solrIndex/foobar/foobar_shard25_replica_t49/data/index.20260215100909238/_55i.nvd\")
>> [this may be caused by lack of enough unfragmented virtual address
>> space or too restrictive virtual memory limits enforced by the
>> operating system, preventing us to map a chunk of 2162259 bytes
>>
>> It throws these memory issues from here:
>> https://github.com/apache/solr/blob/a36c3b511dbd56e929d93f2b2ed7cf94adb436e9/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java#L1303
>>
>>
>> Looking at Lucene, one difference that Lucene 9.12 didn't have is
>> this change relating to ReadOnceHint? Not sure if this makes a
>> difference.https://github.com/apache/lucene/blame/main/lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java#L328
>>
>> From:dev@solr.apache.org At: 02/19/26 12:02:57
>> UTC-5:00To:dev@solr.apache.org
>> Cc:uwe@thetaphi.de
>> Subject: Re: FYI for those trying Java 21
>>
>> Fantastic! PR welcome ;-)
>>
>> If you don't get to it, someone will.
>>
>> While what you suggest looks totally reasonable, do you really think
>> this
>> is likely to fix the matter? We see similar errors for other files
>> like a
>> ".tim" file.
>>
>> On Thu, Feb 19, 2026 at 8:48 AM Uwe Schindler via
>> dev<de...@solr.apache.org>
>> wrote:
>>
>>> i think the issue is here:
>>>
>>>
>> https://github.com/apache/solr/blob/b71c872c35b1a12551e3d9b832c1ef9c40a1ea0a/sol
>>
>> r/core/src/java/org/apache/solr/core/SolrCore.java#L443
>>> this should use IOContext.READ_ONCE
>>>
>>> uwe
>>>
>>> Am 19.02.2026 um 14:09 schrieb Uwe Schindler via dev:
>>>> Oh I see, it is "index.properties". This is not a Lucene file. I think
>>>> when opening that file it should pass READ_ONCE as IOContext to get
>>>> rid of that issue.
>>>>
>>>> Uwe
>>>>
>>>> Am 19.02.2026 um 14:05 schrieb Uwe Schindler via dev:
>>>>> Hi,
>>>>>
>>>>> I think the problem here is that not all fixes around that use-case
>>>>> were applied to Lucene 9. The problem here is that in Lucene 9 we do
>>>>> not open all files in "read-once" mode, especially the segment infos
>>>>> files. The error message about 78 bytes not beeing able to be mapped
>>>>> looks really like the segment infos file, we don't have any other
>>>>> files so small. This is no longer an issue in Lucene 10.
>>>>>
>>>>> Uwe
>>>>>
>>>>> Am 18.02.2026 um 17:40 schrieb David Smiley:
>>>>>> The same boolean setting was useful as a mitigation for the
>>>>>> following
>>>>>> error, totally different business / search cluster -- Solr 9.10,
>>>>>> Java 21,
>>>>>> did not happen with Java 17 with identical search cluster:
>>>>>>
>>>>>>> Caused by: java.io.IOException: Map failed:
>>>>>>>
>> MemorySegmentIndexInput(path="/var/solr/REDACTED_PATH/REDACTED_COLLECTION_shard39_replica_t77/data/index.properties")
>>
>>>>>>> [this may be caused by lack of enough unfragmented virtual address
>>>>>>> space or
>>>>>>> too restrictive virtual memory limits enforced by the operating
>>>>>>> system,
>>>>>>> preventing us to map a chunk of 78 bytes. Please review 'ulimit
>>>>>>> -v',
>>>>>>> 'ulimit -m' (both should return 'unlimited'), and 'sysctl
>>>>>>> vm.max_map_count'
>>>>>> Reminder: the setting doesn't exist in Solr 10; a proper solution
>>>>>> must be
>>>>>> found for a Solr 10 upgrade, assuming the problem re-appears. I'm
>>>>>> somewhat
>>>>>> optimistic a further Java upgrade (to say 25) would be helpful,
>>>>>> albeit I
>>>>>> haven't investigated that yet.
>>>>>>
>>>>>>
>>>>>> On Wed, Aug 21, 2024 at 1:23 PM David Smiley<ds...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> For those trying Java 21 with Solr, if you find the performance
>>>>>>> to be
>>>>>>> bad then try setting
>>>>>>> -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
>>>>>>> This made a huge difference for me on our Solr nodes at work.
>>>>>>>
>>>>>>> I filed the following ticket to Solr that ought to avoid the JDK
>>>>>>> performance regression if done:
>>>>>>> https://issues.apache.org/jira/browse/SOLR-17375
>>>>>>>
>>>>>>> ~ David Smiley
>>>>>>> Apache Lucene/Solr Search Developer
>>>>>>> http://www.linkedin.com/in/davidwsmiley
>>>>>>>
>>> --
>>> Uwe Schindler
>>> Achterdiek 19, D-28357 Bremen
>>> https://www.thetaphi.de
>>> eMail:uwe@thetaphi.de
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail:dev-unsubscribe@solr.apache.org
>>> For additional commands, e-mail:dev-help@solr.apache.org
>>>
>>>
>>
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail:uwe@thetaphi.de
Re: FYI for those trying Java 21
Posted by Uwe Schindler via dev <de...@solr.apache.org>.
Hi,
yes this looks a big problem. The functionality and heuristics for
grouping arenas in MMapDirectory ist made for the usual index file
names. Those filenames do not conform to the standard pattern of Lucene
and therefore cannot be grouped by segment number and it also is not
marked as metadata file. Basically what happens is that files in Lucene
are opened once and to spare system resources and additional overhead
while closing arenas is to group those together. The inappropiate named
files get a new group on each open and they are not released in time and
consume resources. MMapDir cant handle the files correctly.
The problem with Lucene 9 is that it reserves slots for 1024 groups by
default per index and reopening the same file over and over keeps those
groups for much longer than expected. This can be mitigated by another
system property (see MMapDirectory javadocs) to limit the number of
groups to 64 or even lower (this was done in Lucene 10). The limit 1024
was way too large as every index can have up to 1024 arenas open at same
time. The number of mapping allowed by the kernel (unless tuned) is only
65536 arenas / maps. So once you have 64 indexes and reopen the file
over and over while keeping previous mmaps, you run out of mappings in
the kernel.
if you see this error for other files this is just a side effect of the
open same file in a new group, at some point you run out of kernel
resources after 64 indexes.
Quick workaround:
https://lucene.apache.org/core/9_12_0/core/org/apache/lucene/store/MMapDirectory.html#NO_GROUPING
This restores behaviour of around Lucene 9.10 where all files are closed
and all their mappings not cached.
Or set following sysprop (defaults 1024) to much lower values (I'd
recommend 16 if you have many indexes on same instance):
org.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits
(with the expense of slowdown on closing IndexInputs).
Uwe
Am 19.02.2026 um 18:21 schrieb Matthew Biscocho (BLOOMBERG/ 120 PARK):
> Thanks Uwe for the info.
>
> We ran into this error in a bunch of different ways from Solr. I did some digging and we ran into this with other places that should probably have this change as well for example the replication.properties file:
>
> https://github.com/apache/solr/blob/a36c3b511dbd56e929d93f2b2ed7cf94adb436e9/solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java#L1189
>
>
> Ran into it here that also include actual segments that probably should be READONCE probably:
> https://github.com/apache/solr/blob/a36c3b511dbd56e929d93f2b2ed7cf94adb436e9/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java#L1367
> But this issue still occurred with segment files that did do a READONCE
>
> java.io.IOException: Map failed: MemorySegmentIndexInput(path=\"/var/solrIndex/foobar/foobar_shard25_replica_t49/data/index.20260215100909238/_55i.nvd\") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 2162259 bytes
>
> It throws these memory issues from here:
> https://github.com/apache/solr/blob/a36c3b511dbd56e929d93f2b2ed7cf94adb436e9/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java#L1303
>
> Looking at Lucene, one difference that Lucene 9.12 didn't have is this change relating to ReadOnceHint? Not sure if this makes a difference.https://github.com/apache/lucene/blame/main/lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java#L328
>
> From:dev@solr.apache.org At: 02/19/26 12:02:57 UTC-5:00To:dev@solr.apache.org
> Cc:uwe@thetaphi.de
> Subject: Re: FYI for those trying Java 21
>
> Fantastic! PR welcome ;-)
>
> If you don't get to it, someone will.
>
> While what you suggest looks totally reasonable, do you really think this
> is likely to fix the matter? We see similar errors for other files like a
> ".tim" file.
>
> On Thu, Feb 19, 2026 at 8:48 AM Uwe Schindler via dev<de...@solr.apache.org>
> wrote:
>
>> i think the issue is here:
>>
>>
> https://github.com/apache/solr/blob/b71c872c35b1a12551e3d9b832c1ef9c40a1ea0a/sol
> r/core/src/java/org/apache/solr/core/SolrCore.java#L443
>> this should use IOContext.READ_ONCE
>>
>> uwe
>>
>> Am 19.02.2026 um 14:09 schrieb Uwe Schindler via dev:
>>> Oh I see, it is "index.properties". This is not a Lucene file. I think
>>> when opening that file it should pass READ_ONCE as IOContext to get
>>> rid of that issue.
>>>
>>> Uwe
>>>
>>> Am 19.02.2026 um 14:05 schrieb Uwe Schindler via dev:
>>>> Hi,
>>>>
>>>> I think the problem here is that not all fixes around that use-case
>>>> were applied to Lucene 9. The problem here is that in Lucene 9 we do
>>>> not open all files in "read-once" mode, especially the segment infos
>>>> files. The error message about 78 bytes not beeing able to be mapped
>>>> looks really like the segment infos file, we don't have any other
>>>> files so small. This is no longer an issue in Lucene 10.
>>>>
>>>> Uwe
>>>>
>>>> Am 18.02.2026 um 17:40 schrieb David Smiley:
>>>>> The same boolean setting was useful as a mitigation for the following
>>>>> error, totally different business / search cluster -- Solr 9.10,
>>>>> Java 21,
>>>>> did not happen with Java 17 with identical search cluster:
>>>>>
>>>>>> Caused by: java.io.IOException: Map failed:
>>>>>>
> MemorySegmentIndexInput(path="/var/solr/REDACTED_PATH/REDACTED_COLLECTION_shard39_replica_t77/data/index.properties")
>>>>>> [this may be caused by lack of enough unfragmented virtual address
>>>>>> space or
>>>>>> too restrictive virtual memory limits enforced by the operating
>>>>>> system,
>>>>>> preventing us to map a chunk of 78 bytes. Please review 'ulimit -v',
>>>>>> 'ulimit -m' (both should return 'unlimited'), and 'sysctl
>>>>>> vm.max_map_count'
>>>>> Reminder: the setting doesn't exist in Solr 10; a proper solution
>>>>> must be
>>>>> found for a Solr 10 upgrade, assuming the problem re-appears. I'm
>>>>> somewhat
>>>>> optimistic a further Java upgrade (to say 25) would be helpful,
>>>>> albeit I
>>>>> haven't investigated that yet.
>>>>>
>>>>>
>>>>> On Wed, Aug 21, 2024 at 1:23 PM David Smiley<ds...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> For those trying Java 21 with Solr, if you find the performance to be
>>>>>> bad then try setting
>>>>>> -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
>>>>>> This made a huge difference for me on our Solr nodes at work.
>>>>>>
>>>>>> I filed the following ticket to Solr that ought to avoid the JDK
>>>>>> performance regression if done:
>>>>>> https://issues.apache.org/jira/browse/SOLR-17375
>>>>>>
>>>>>> ~ David Smiley
>>>>>> Apache Lucene/Solr Search Developer
>>>>>> http://www.linkedin.com/in/davidwsmiley
>>>>>>
>> --
>> Uwe Schindler
>> Achterdiek 19, D-28357 Bremen
>> https://www.thetaphi.de
>> eMail:uwe@thetaphi.de
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:dev-unsubscribe@solr.apache.org
>> For additional commands, e-mail:dev-help@solr.apache.org
>>
>>
>
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail:uwe@thetaphi.de
Re: FYI for those trying Java 21
Posted by "Matthew Biscocho (BLOOMBERG/ 120 PARK)" <mb...@bloomberg.net>.
Thanks Uwe for the info.
We ran into this error in a bunch of different ways from Solr. I did some digging and we ran into this with other places that should probably have this change as well for example the replication.properties file:
https://github.com/apache/solr/blob/a36c3b511dbd56e929d93f2b2ed7cf94adb436e9/solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java#L1189
Ran into it here that also include actual segments that probably should be READONCE probably:
https://github.com/apache/solr/blob/a36c3b511dbd56e929d93f2b2ed7cf94adb436e9/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java#L1367
But this issue still occurred with segment files that did do a READONCE
java.io.IOException: Map failed: MemorySegmentIndexInput(path=\"/var/solrIndex/foobar/foobar_shard25_replica_t49/data/index.20260215100909238/_55i.nvd\") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 2162259 bytes
It throws these memory issues from here:
https://github.com/apache/solr/blob/a36c3b511dbd56e929d93f2b2ed7cf94adb436e9/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java#L1303
Looking at Lucene, one difference that Lucene 9.12 didn't have is this change relating to ReadOnceHint? Not sure if this makes a difference. https://github.com/apache/lucene/blame/main/lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java#L328
From: dev@solr.apache.org At: 02/19/26 12:02:57 UTC-5:00To: dev@solr.apache.org
Cc: uwe@thetaphi.de
Subject: Re: FYI for those trying Java 21
Fantastic! PR welcome ;-)
If you don't get to it, someone will.
While what you suggest looks totally reasonable, do you really think this
is likely to fix the matter? We see similar errors for other files like a
".tim" file.
On Thu, Feb 19, 2026 at 8:48 AM Uwe Schindler via dev <de...@solr.apache.org>
wrote:
> i think the issue is here:
>
>
https://github.com/apache/solr/blob/b71c872c35b1a12551e3d9b832c1ef9c40a1ea0a/sol
r/core/src/java/org/apache/solr/core/SolrCore.java#L443
>
> this should use IOContext.READ_ONCE
>
> uwe
>
> Am 19.02.2026 um 14:09 schrieb Uwe Schindler via dev:
> > Oh I see, it is "index.properties". This is not a Lucene file. I think
> > when opening that file it should pass READ_ONCE as IOContext to get
> > rid of that issue.
> >
> > Uwe
> >
> > Am 19.02.2026 um 14:05 schrieb Uwe Schindler via dev:
> >> Hi,
> >>
> >> I think the problem here is that not all fixes around that use-case
> >> were applied to Lucene 9. The problem here is that in Lucene 9 we do
> >> not open all files in "read-once" mode, especially the segment infos
> >> files. The error message about 78 bytes not beeing able to be mapped
> >> looks really like the segment infos file, we don't have any other
> >> files so small. This is no longer an issue in Lucene 10.
> >>
> >> Uwe
> >>
> >> Am 18.02.2026 um 17:40 schrieb David Smiley:
> >>> The same boolean setting was useful as a mitigation for the following
> >>> error, totally different business / search cluster -- Solr 9.10,
> >>> Java 21,
> >>> did not happen with Java 17 with identical search cluster:
> >>>
> >>>> Caused by: java.io.IOException: Map failed:
> >>>>
>
MemorySegmentIndexInput(path="/var/solr/REDACTED_PATH/REDACTED_COLLECTION_shard39_replica_t77/data/index.properties")
>
> >>>>
> >>>> [this may be caused by lack of enough unfragmented virtual address
> >>>> space or
> >>>> too restrictive virtual memory limits enforced by the operating
> >>>> system,
> >>>> preventing us to map a chunk of 78 bytes. Please review 'ulimit -v',
> >>>> 'ulimit -m' (both should return 'unlimited'), and 'sysctl
> >>>> vm.max_map_count'
> >>>
> >>> Reminder: the setting doesn't exist in Solr 10; a proper solution
> >>> must be
> >>> found for a Solr 10 upgrade, assuming the problem re-appears. I'm
> >>> somewhat
> >>> optimistic a further Java upgrade (to say 25) would be helpful,
> >>> albeit I
> >>> haven't investigated that yet.
> >>>
> >>>
> >>> On Wed, Aug 21, 2024 at 1:23 PM David Smiley <ds...@apache.org>
> >>> wrote:
> >>>
> >>>> For those trying Java 21 with Solr, if you find the performance to be
> >>>> bad then try setting
> >>>> -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
> >>>> This made a huge difference for me on our Solr nodes at work.
> >>>>
> >>>> I filed the following ticket to Solr that ought to avoid the JDK
> >>>> performance regression if done:
> >>>> https://issues.apache.org/jira/browse/SOLR-17375
> >>>>
> >>>> ~ David Smiley
> >>>> Apache Lucene/Solr Search Developer
> >>>> http://www.linkedin.com/in/davidwsmiley
> >>>>
> --
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
Re: FYI for those trying Java 21
Posted by David Smiley <ds...@apache.org>.
Fantastic! PR welcome ;-)
If you don't get to it, someone will.
While what you suggest looks totally reasonable, do you really think this
is likely to fix the matter? We see similar errors for other files like a
".tim" file.
On Thu, Feb 19, 2026 at 8:48 AM Uwe Schindler via dev <de...@solr.apache.org>
wrote:
> i think the issue is here:
>
> https://github.com/apache/solr/blob/b71c872c35b1a12551e3d9b832c1ef9c40a1ea0a/solr/core/src/java/org/apache/solr/core/SolrCore.java#L443
>
> this should use IOContext.READ_ONCE
>
> uwe
>
> Am 19.02.2026 um 14:09 schrieb Uwe Schindler via dev:
> > Oh I see, it is "index.properties". This is not a Lucene file. I think
> > when opening that file it should pass READ_ONCE as IOContext to get
> > rid of that issue.
> >
> > Uwe
> >
> > Am 19.02.2026 um 14:05 schrieb Uwe Schindler via dev:
> >> Hi,
> >>
> >> I think the problem here is that not all fixes around that use-case
> >> were applied to Lucene 9. The problem here is that in Lucene 9 we do
> >> not open all files in "read-once" mode, especially the segment infos
> >> files. The error message about 78 bytes not beeing able to be mapped
> >> looks really like the segment infos file, we don't have any other
> >> files so small. This is no longer an issue in Lucene 10.
> >>
> >> Uwe
> >>
> >> Am 18.02.2026 um 17:40 schrieb David Smiley:
> >>> The same boolean setting was useful as a mitigation for the following
> >>> error, totally different business / search cluster -- Solr 9.10,
> >>> Java 21,
> >>> did not happen with Java 17 with identical search cluster:
> >>>
> >>>> Caused by: java.io.IOException: Map failed:
> >>>>
> MemorySegmentIndexInput(path="/var/solr/REDACTED_PATH/REDACTED_COLLECTION_shard39_replica_t77/data/index.properties")
>
> >>>>
> >>>> [this may be caused by lack of enough unfragmented virtual address
> >>>> space or
> >>>> too restrictive virtual memory limits enforced by the operating
> >>>> system,
> >>>> preventing us to map a chunk of 78 bytes. Please review 'ulimit -v',
> >>>> 'ulimit -m' (both should return 'unlimited'), and 'sysctl
> >>>> vm.max_map_count'
> >>>
> >>> Reminder: the setting doesn't exist in Solr 10; a proper solution
> >>> must be
> >>> found for a Solr 10 upgrade, assuming the problem re-appears. I'm
> >>> somewhat
> >>> optimistic a further Java upgrade (to say 25) would be helpful,
> >>> albeit I
> >>> haven't investigated that yet.
> >>>
> >>>
> >>> On Wed, Aug 21, 2024 at 1:23 PM David Smiley <ds...@apache.org>
> >>> wrote:
> >>>
> >>>> For those trying Java 21 with Solr, if you find the performance to be
> >>>> bad then try setting
> >>>> -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
> >>>> This made a huge difference for me on our Solr nodes at work.
> >>>>
> >>>> I filed the following ticket to Solr that ought to avoid the JDK
> >>>> performance regression if done:
> >>>> https://issues.apache.org/jira/browse/SOLR-17375
> >>>>
> >>>> ~ David Smiley
> >>>> Apache Lucene/Solr Search Developer
> >>>> http://www.linkedin.com/in/davidwsmiley
> >>>>
> --
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
Re: FYI for those trying Java 21
Posted by Uwe Schindler via dev <de...@solr.apache.org>.
i think the issue is here:
https://github.com/apache/solr/blob/b71c872c35b1a12551e3d9b832c1ef9c40a1ea0a/solr/core/src/java/org/apache/solr/core/SolrCore.java#L443
this should use IOContext.READ_ONCE
uwe
Am 19.02.2026 um 14:09 schrieb Uwe Schindler via dev:
> Oh I see, it is "index.properties". This is not a Lucene file. I think
> when opening that file it should pass READ_ONCE as IOContext to get
> rid of that issue.
>
> Uwe
>
> Am 19.02.2026 um 14:05 schrieb Uwe Schindler via dev:
>> Hi,
>>
>> I think the problem here is that not all fixes around that use-case
>> were applied to Lucene 9. The problem here is that in Lucene 9 we do
>> not open all files in "read-once" mode, especially the segment infos
>> files. The error message about 78 bytes not beeing able to be mapped
>> looks really like the segment infos file, we don't have any other
>> files so small. This is no longer an issue in Lucene 10.
>>
>> Uwe
>>
>> Am 18.02.2026 um 17:40 schrieb David Smiley:
>>> The same boolean setting was useful as a mitigation for the following
>>> error, totally different business / search cluster -- Solr 9.10,
>>> Java 21,
>>> did not happen with Java 17 with identical search cluster:
>>>
>>>> Caused by: java.io.IOException: Map failed:
>>>> MemorySegmentIndexInput(path="/var/solr/REDACTED_PATH/REDACTED_COLLECTION_shard39_replica_t77/data/index.properties")
>>>>
>>>> [this may be caused by lack of enough unfragmented virtual address
>>>> space or
>>>> too restrictive virtual memory limits enforced by the operating
>>>> system,
>>>> preventing us to map a chunk of 78 bytes. Please review 'ulimit -v',
>>>> 'ulimit -m' (both should return 'unlimited'), and 'sysctl
>>>> vm.max_map_count'
>>>
>>> Reminder: the setting doesn't exist in Solr 10; a proper solution
>>> must be
>>> found for a Solr 10 upgrade, assuming the problem re-appears. I'm
>>> somewhat
>>> optimistic a further Java upgrade (to say 25) would be helpful,
>>> albeit I
>>> haven't investigated that yet.
>>>
>>>
>>> On Wed, Aug 21, 2024 at 1:23 PM David Smiley <ds...@apache.org>
>>> wrote:
>>>
>>>> For those trying Java 21 with Solr, if you find the performance to be
>>>> bad then try setting
>>>> -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
>>>> This made a huge difference for me on our Solr nodes at work.
>>>>
>>>> I filed the following ticket to Solr that ought to avoid the JDK
>>>> performance regression if done:
>>>> https://issues.apache.org/jira/browse/SOLR-17375
>>>>
>>>> ~ David Smiley
>>>> Apache Lucene/Solr Search Developer
>>>> http://www.linkedin.com/in/davidwsmiley
>>>>
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: uwe@thetaphi.de
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: FYI for those trying Java 21
Posted by Uwe Schindler via dev <de...@solr.apache.org>.
Oh I see, it is "index.properties". This is not a Lucene file. I think
when opening that file it should pass READ_ONCE as IOContext to get rid
of that issue.
Uwe
Am 19.02.2026 um 14:05 schrieb Uwe Schindler via dev:
> Hi,
>
> I think the problem here is that not all fixes around that use-case
> were applied to Lucene 9. The problem here is that in Lucene 9 we do
> not open all files in "read-once" mode, especially the segment infos
> files. The error message about 78 bytes not beeing able to be mapped
> looks really like the segment infos file, we don't have any other
> files so small. This is no longer an issue in Lucene 10.
>
> Uwe
>
> Am 18.02.2026 um 17:40 schrieb David Smiley:
>> The same boolean setting was useful as a mitigation for the following
>> error, totally different business / search cluster -- Solr 9.10, Java
>> 21,
>> did not happen with Java 17 with identical search cluster:
>>
>>> Caused by: java.io.IOException: Map failed:
>>> MemorySegmentIndexInput(path="/var/solr/REDACTED_PATH/REDACTED_COLLECTION_shard39_replica_t77/data/index.properties")
>>>
>>> [this may be caused by lack of enough unfragmented virtual address
>>> space or
>>> too restrictive virtual memory limits enforced by the operating system,
>>> preventing us to map a chunk of 78 bytes. Please review 'ulimit -v',
>>> 'ulimit -m' (both should return 'unlimited'), and 'sysctl
>>> vm.max_map_count'
>>
>> Reminder: the setting doesn't exist in Solr 10; a proper solution
>> must be
>> found for a Solr 10 upgrade, assuming the problem re-appears. I'm
>> somewhat
>> optimistic a further Java upgrade (to say 25) would be helpful, albeit I
>> haven't investigated that yet.
>>
>>
>> On Wed, Aug 21, 2024 at 1:23 PM David Smiley <ds...@apache.org> wrote:
>>
>>> For those trying Java 21 with Solr, if you find the performance to be
>>> bad then try setting
>>> -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
>>> This made a huge difference for me on our Solr nodes at work.
>>>
>>> I filed the following ticket to Solr that ought to avoid the JDK
>>> performance regression if done:
>>> https://issues.apache.org/jira/browse/SOLR-17375
>>>
>>> ~ David Smiley
>>> Apache Lucene/Solr Search Developer
>>> http://www.linkedin.com/in/davidwsmiley
>>>
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: uwe@thetaphi.de
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: FYI for those trying Java 21
Posted by Uwe Schindler via dev <de...@solr.apache.org>.
Hi,
I think the problem here is that not all fixes around that use-case were
applied to Lucene 9. The problem here is that in Lucene 9 we do not open
all files in "read-once" mode, especially the segment infos files. The
error message about 78 bytes not beeing able to be mapped looks really
like the segment infos file, we don't have any other files so small.
This is no longer an issue in Lucene 10.
Uwe
Am 18.02.2026 um 17:40 schrieb David Smiley:
> The same boolean setting was useful as a mitigation for the following
> error, totally different business / search cluster -- Solr 9.10, Java 21,
> did not happen with Java 17 with identical search cluster:
>
>> Caused by: java.io.IOException: Map failed:
>> MemorySegmentIndexInput(path="/var/solr/REDACTED_PATH/REDACTED_COLLECTION_shard39_replica_t77/data/index.properties")
>> [this may be caused by lack of enough unfragmented virtual address space or
>> too restrictive virtual memory limits enforced by the operating system,
>> preventing us to map a chunk of 78 bytes. Please review 'ulimit -v',
>> 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max_map_count'
>
> Reminder: the setting doesn't exist in Solr 10; a proper solution must be
> found for a Solr 10 upgrade, assuming the problem re-appears. I'm somewhat
> optimistic a further Java upgrade (to say 25) would be helpful, albeit I
> haven't investigated that yet.
>
>
> On Wed, Aug 21, 2024 at 1:23 PM David Smiley <ds...@apache.org> wrote:
>
>> For those trying Java 21 with Solr, if you find the performance to be
>> bad then try setting
>> -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
>> This made a huge difference for me on our Solr nodes at work.
>>
>> I filed the following ticket to Solr that ought to avoid the JDK
>> performance regression if done:
>> https://issues.apache.org/jira/browse/SOLR-17375
>>
>> ~ David Smiley
>> Apache Lucene/Solr Search Developer
>> http://www.linkedin.com/in/davidwsmiley
>>
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: uwe@thetaphi.de
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: FYI for those trying Java 21
Posted by David Smiley <ds...@apache.org>.
The same boolean setting was useful as a mitigation for the following
error, totally different business / search cluster -- Solr 9.10, Java 21,
did not happen with Java 17 with identical search cluster:
> Caused by: java.io.IOException: Map failed:
> MemorySegmentIndexInput(path="/var/solr/REDACTED_PATH/REDACTED_COLLECTION_shard39_replica_t77/data/index.properties")
> [this may be caused by lack of enough unfragmented virtual address space or
> too restrictive virtual memory limits enforced by the operating system,
> preventing us to map a chunk of 78 bytes. Please review 'ulimit -v',
> 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max_map_count'
Reminder: the setting doesn't exist in Solr 10; a proper solution must be
found for a Solr 10 upgrade, assuming the problem re-appears. I'm somewhat
optimistic a further Java upgrade (to say 25) would be helpful, albeit I
haven't investigated that yet.
On Wed, Aug 21, 2024 at 1:23 PM David Smiley <ds...@apache.org> wrote:
> For those trying Java 21 with Solr, if you find the performance to be
> bad then try setting
> -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
> This made a huge difference for me on our Solr nodes at work.
>
> I filed the following ticket to Solr that ought to avoid the JDK
> performance regression if done:
> https://issues.apache.org/jira/browse/SOLR-17375
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
Re: FYI for those trying Java 21
Posted by David Smiley <ds...@apache.org>.
The same boolean setting was useful as a mitigation for the following
error, totally different business / search cluster -- Solr 9.10, Java 21,
did not happen with Java 17 with identical search cluster:
> Caused by: java.io.IOException: Map failed:
> MemorySegmentIndexInput(path="/var/solr/REDACTED_PATH/REDACTED_COLLECTION_shard39_replica_t77/data/index.properties")
> [this may be caused by lack of enough unfragmented virtual address space or
> too restrictive virtual memory limits enforced by the operating system,
> preventing us to map a chunk of 78 bytes. Please review 'ulimit -v',
> 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max_map_count'
Reminder: the setting doesn't exist in Solr 10; a proper solution must be
found for a Solr 10 upgrade, assuming the problem re-appears. I'm somewhat
optimistic a further Java upgrade (to say 25) would be helpful, albeit I
haven't investigated that yet.
On Wed, Aug 21, 2024 at 1:23 PM David Smiley <ds...@apache.org> wrote:
> For those trying Java 21 with Solr, if you find the performance to be
> bad then try setting
> -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
> This made a huge difference for me on our Solr nodes at work.
>
> I filed the following ticket to Solr that ought to avoid the JDK
> performance regression if done:
> https://issues.apache.org/jira/browse/SOLR-17375
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
Re:FYI for those trying Java 21
Posted by "Kevin Liang (BLOOMBERG/ 919 3RD A)" <kl...@bloomberg.net>.
Thanks David for the tip. We'll test with/without the flag and report back our findings.
-Kevin
From: users@solr.apache.org At: 08/21/24 13:24:10 UTC-4:00To: dev@solr.apache.org, users@solr.apache.org
Subject: FYI for those trying Java 21
For those trying Java 21 with Solr, if you find the performance to be
bad then try setting
-Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
This made a huge difference for me on our Solr nodes at work.
I filed the following ticket to Solr that ought to avoid the JDK
performance regression if done:
https://issues.apache.org/jira/browse/SOLR-17375
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley