You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "ChrisHegarty (via GitHub)" <gi...@apache.org> on 2025/03/26 17:34:56 UTC
[I] Examine the affects of MADV_RANDOM when MGLRU is enabled in Linux kernel [lucene]
ChrisHegarty opened a new issue, #14408:
URL: https://github.com/apache/lucene/issues/14408
With the relatively recent capability to call `madvise` in Lucene, we've started to use `MADV_RANDOM` in several places where it makes conceptual sense, e.g. for accessing vector data when navigating the graph. The memory access is truly random, but we've seen several reports of performance regressions that appear as a result of this.
Of particular concern is the interaction of `MADV_RANDOM` with Multi-Gen LRU [1]. From my reading of the code, and someone please correct me, the semantics of `MADV_RANDOM` has changed in the kernel with MGLRU, and results in pages being proactively reclaimed more eagerly, even when there is no memory pressure. Specifically after https://github.com/torvalds/linux/commit/8788f6781486769d9598dcaedc3fe0eb12fc3e59.
This Elasticsearch issue has the more of the lower-level details, https://github.com/elastic/elasticsearch/issues/124499.
This issue may also have some connection, https://github.com/apache/lucene/issues/14281.
I opened this issue to help facilitate a discussion and hopefully converge on a potential direction to mitigate the possibility of performance regressions. For example, one possible mitigation would be to expose the `ReadAdivce` that will be used as part of the API, so that callers can have more fine-grained control over whether or not to use `MADV_RANDOM`.
[1] https://docs.kernel.org/admin-guide/mm/multigen_lru.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine the affects of MADV_RANDOM when MGLRU is enabled in Linux kernel [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on issue #14408:
URL: https://github.com/apache/lucene/issues/14408#issuecomment-2755385165
A better default would also be possible, I think, if individual codecs such as vectors, did not set this directly, but instead allowed the Directory class to be in control.
User/Directory is only partially in control now, there is a `org.apache.lucene.store.defaultReadAdvice` system property, but then this codec just ignores that, which I don't think is good. So a user has to write custom Directory and mess around with IOContext to correct the issue, which isn't obvious.
Let the defaults be as smart as they need. Maybe check `/sys/kernel/mm/lru_gen/enabled` as part of the decision-making! But IMO let the user have the final say, in an easy way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine the affects of MADV_RANDOM when MGLRU is enabled in Linux kernel [lucene]
Posted by "vigyasharma (via GitHub)" <gi...@apache.org>.
vigyasharma commented on issue #14408:
URL: https://github.com/apache/lucene/issues/14408#issuecomment-2828222009
Should we target a revert of the default ReadAdvice.RANDOM setting, for the 10.2.1 RC that's currently being evaluated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine the affects of MADV_RANDOM when MGLRU is enabled in Linux kernel [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on issue #14408:
URL: https://github.com/apache/lucene/issues/14408#issuecomment-2830144654
> +1 to work towards this more general fix. But, sheesh, it looks so complicated, depending on hot vs cold use case, preloading or not, which part of the Lucene index (KNN, terms, postings), Linux kernel versions, ... in the mean time I think we should revert back to `NORMAL`/`DEFAULT` as Lucene's default...
Yes, my argument is that it is complicated, and lucene needs to get out of the business of it.
We can flip-flop on this setting over and over again, and each time some users will experience regressions and others will get happy.
I don't want to see this, it is just more of the same. Lucene needs to get out of the business of doing it.
But if you want a quick fix: I'd be in favor of a PR that removes all madvise/preloading/otherwise from lucene. Let it be the "user's decision" on this shit, the different use-cases and platforms are too different, lucene cannot have "defaults" in java code and pretend like thats gonna work well across all these various use-cases and platforms. it does not work. and for users that care and want this stuff, hopefully it gets easier for them once #14422 lands.
This way its not constant regressing back and forth because of settings flipflopping
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine the affects of MADV_RANDOM when MGLRU is enabled in Linux kernel [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on issue #14408:
URL: https://github.com/apache/lucene/issues/14408#issuecomment-2755325880
> For example, one possible mitigation would be to expose the `ReadAdivce` that will be used as part of the API, so that callers can have more fine-grained control over whether or not to use `MADV_RANDOM`.
This is already supported: it is the `Directory` api which is used to customize such behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine the affects of MADV_RANDOM when MGLRU is enabled in Linux kernel [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on issue #14408:
URL: https://github.com/apache/lucene/issues/14408#issuecomment-2828267119
IMO it would be better to make a 10.2.2 if you want to do that? Especially since I don't think it would be a trivial change: I'm concerned that changing just the "default" would have any benefit, since it is hardcoded in other places in codecs. AFAIK #14422 is working on fixing that "real problem".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine the affects of MADV_RANDOM when MGLRU is enabled in Linux kernel [lucene]
Posted by "jimczi (via GitHub)" <gi...@apache.org>.
jimczi commented on issue #14408:
URL: https://github.com/apache/lucene/issues/14408#issuecomment-2755375551
I believe the question is whether we need to reconsider our assumptions when defaulting to random read advice in the current code. With the linked change, using `MADV_RANDOM` will exclude pages from the LRU list, but the original intent was simply to reduce the read-ahead size.
We explicitly use random advice in the vector format and FST files, which should, by default, benefit from LRU. Users should not be required to make changes to achieve the correct behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine the affects of MADV_RANDOM when MGLRU is enabled in Linux kernel [lucene]
Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on issue #14408:
URL: https://github.com/apache/lucene/issues/14408#issuecomment-3167479398
I opened the following PR to revert the read advice to NORMAL, see #15040
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine the affects of MADV_RANDOM when MGLRU is enabled in Linux kernel [lucene]
Posted by "jimczi (via GitHub)" <gi...@apache.org>.
jimczi commented on issue #14408:
URL: https://github.com/apache/lucene/issues/14408#issuecomment-2755462640
> Let the defaults be as smart as they need. Maybe check /sys/kernel/mm/lru_gen/enabled as part of the decision-making! But IMO let the user have the final say, in an easy way.
++, the normal path for users to override is to use the system property or to write a custom directory if they want the full control. We should remove the overrides in the vector search codec. We default to random anyway.
> Maybe check /sys/kernel/mm/lru_gen/enabled as part of the decision-making!
The Linux change targets both MGLRU and normal LRU. The impact is more pronounced in MGLRU, as page reclamation is more aggressive there. However, the semantic change for this advice is the same in both cases. In the latest kernels, using `MADV_RANDOM` does not mark the page as accessed, regardless of whether MGLRU is in use. That's a big shift of semantic for our default read advice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine the affects of MADV_RANDOM when MGLRU is enabled in Linux kernel [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on issue #14408:
URL: https://github.com/apache/lucene/issues/14408#issuecomment-2755662166
> The Linux change targets both MGLRU and normal LRU. The impact is more pronounced in MGLRU, as page reclamation is more aggressive there. However, the semantic change for this advice is the same in both cases. In the latest kernels, using `MADV_RANDOM` does not mark the page as accessed, regardless of whether MGLRU is in use. That's a big shift of semantic for our default read advice.
Easy argument to change the default to `NORMAL`.
Another idea here, would be to add something like a `ReadAdviceDirectory` that makes use of `FilterDirectory` to let the user configure these things without using a system property. It would be nice if they could specify to PRELOAD *.vec files, or RANDOM *.fdt files, if they are interested in tweaking performance. Then maybe it is ok for defaults to be NORMAL (simple) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine the affects of MADV_RANDOM when MGLRU is enabled in Linux kernel [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #14408:
URL: https://github.com/apache/lucene/issues/14408#issuecomment-2830109470
> > The Linux change targets both MGLRU and normal LRU. The impact is more pronounced in MGLRU, as page reclamation is more aggressive there. However, the semantic change for this advice is the same in both cases. In the latest kernels, using `MADV_RANDOM` does not mark the page as accessed, regardless of whether MGLRU is in use. That's a big shift of semantic for our default read advice.
>
> Easy argument to change the default to `NORMAL`.
+1 to go back to `NORMAL` as default, until we can better understand the regressions we (OpenSearch users, Elasticsearch users, and Amazon product search (my team)) are seeing with `MADV_RANDOM`.
I think `MADV_RANDOM` can also be harmful for "hot" (index expected to mostly fit in RAM) use cases.
For our service (Amazon product search), which is mostly hot, we had to hard-override back to `IOContext.DEFAULT` for `.vec` and `.veq` (quantized vectors) in a hackity way (subclass `MMapDirectory` to insert shim (that rewrites the `IOContext`) into `openInput` -- oooh as @jpountz describes at https://github.com/apache/lucene/issues/14348#issuecomment-2730966937, except opposite), in some cases (lighting a new commit point during NRT replication) where we had to turn off `MMapDirectory.setPreload`.
At Lucene's defaults (`MADV_RANDOM` for the KNN vector files) we saw horribly slow warmup of our searchers ... basically, paging in all those vectors one at a time as "real" queries visited the HNSW graph was crazy slow (many minutes) even on crazy fast infra (AWS), whereas letting the OS do its default "thing" (bulk readahead of N pages when a page miss happens?) was much quicker. Much less "page fault amplification".
Benchmarks in luceneutil also hit this -- minutes and minutes of swapping in the HNSW graph (without `.setPreload`) from a fast local SSD, but I think luceneutil is still using Lucene's `IOContext` defaults here.
Actually, if we `MADV_RANDOM` and `.setPreload` to load `.vec`, what is the effect? Does the preloading still work (OS caches/touches all pages, and does mark them as accessed (so they stay cached), despite the `MADV_RANDOM`)? Is it much slower to preload when you `MADV_RANDOM` (though presumably it is sequentially bringing pages in)?
> AFAIK https://github.com/apache/lucene/issues/14422 is working on fixing that "real problem".
+1 to work towards this more general fix. But, sheesh, it looks so complicated, depending on hot vs cold use case, preloading or not, which part of the Lucene index (KNN, terms, postings), Linux kernel versions, ... in the mean time I think we should revert back to `NORMAL`/`DEFAULT` as Lucene's default...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Examine the affects of MADV_RANDOM when MGLRU is enabled in Linux kernel [lucene]
Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty closed issue #14408: Examine the affects of MADV_RANDOM when MGLRU is enabled in Linux kernel
URL: https://github.com/apache/lucene/issues/14408
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org