You are viewing a plain text version of this content. The canonical link for it is here.

Posted to java-user@lucene.apache.org by ba...@oracle.com on 2021/02/23 07:30:30 UTC

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

Hi,-

  I tried MMapDirectory and i allocated as big as index size on my J2EE 
Container but

it only gives me at most 25% speedup and even sometimes a small amount 
of slowdown.

How can i effectively use Lucene indexes in memory?

Best regards


On 12/14/20 6:35 PM, baris.kazar@oracle.com wrote:
> Thanks Robert.
>
> I think these valuable comments need to be placed on javadocs for 
> future references.
>
> i think i am getting enough info for making a decision:
>
> i will use MMapDirectory without setPreload and i hope my index will 
> fit into the RAM.
>
> i plan to post a blog for findings.
>
> Best regards
>
>
> On 12/14/20 5:52 PM, Robert Muir wrote:
>> On Mon, Dec 14, 2020 at 1:59 PM Uwe Schindler <uw...@thetaphi.de> wrote:
>>> Hi,
>>>
>>> as writer of the original bog post, here my comments:
>>>
>>> Yes, MMapDirectory.setPreload() is the feature mentioned in my blog 
>>> post is
>>> to load everything into memory - but that does not guarantee anything!
>>> Still, I would not recommend to use that function, because all it 
>>> does is to
>>> just touch every page of the file, so the linux kernel puts it into 
>>> OS cache
>>> - nothing more; IMHO very ineffective as it slows down openining 
>>> index for a
>>> stupid for-each-page-touch-loop. It will do this with EVERY page, if 
>>> it is
>>> later used or not! So this may take some time until it is done. 
>>> Lateron,
>>> still Lucene needs to open index files, initialize its own data
>>> structures,...
>>>
>>> In general it is much better to open index, with MMAP directory and 
>>> execute
>>> some "sample" queries. This will do exactly the same like the preload
>>> function, but it is more "selective". Parts of the index which are 
>>> not used
>>> won't be touched, and on top, it will also load ALL the required index
>>> structures to heap.
>>>
>> The main purpose of this thing is a fast warming option for random
>> access files such as "i want to warm all my norms in RAM" or "i want
>> to warm all my docvalues in RAM"... really it should only be used with
>> the FileSwitchDirectory for a targeted purpose such as that: it is
>> definitely a waste to set it for your entire index. It is just
>> exposing the 
>> https://docs.oracle.com/javase/7/docs/api/java/nio/MappedByteBuffer.html#load()
>> which first calls madvise(MADV_WILLNEED) and then touches every page.
>> If you want to "warm" an ENTIRE very specific file for a reason like
>> this (e.g. per-doc scoring value, ensuring it will be hot for all
>> docs), it is hard to be more efficient than that.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

Posted by Baris Kazar <ba...@oracle.com>.

So, just cat <Lucene_index_file> will do this.
Thanks
________________________________
From: Robert Muir <rc...@gmail.com>
Sent: Tuesday, February 23, 2021 4:45 PM
To: Baris Kazar <ba...@oracle.com>
Cc: java-user <ja...@lucene.apache.org>
Subject: Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

The preload isn't magical.
It only "reads in the whole file" to get it cached, same as if you did that yourself with 'cat' or 'dd'.
It "warms" the file.

It just does this in an efficient way at the low level to make the warming itself efficient. It madvise()s kernel to announce some read-ahead and then reads the first byte of every mmap'd page (which is enough to fault it in).

At the end of the day it doesn't matter if you wrote a shitty shell script that uses 'dd' to read in each index file and send it to /dev/null, or whether you spent lots of time writing fancy java code to call this preload thing: you get the same result, same end state.

Maybe the preload takes 18 seconds to "warm" the index, vs. your crappy shell script which takes 22 seconds. It is mainly more important for servers and portability (e.g. it will work fine on windows, but obviously will not call madvise).

On Tue, Feb 23, 2021 at 4:18 PM <ba...@oracle.com>> wrote:

Thanks again, Robert. Could you please explain "preload"? Which functionality is that? we discussed in this thread before about a preload.

Is there a Lucene url / site that i can look at for preload?

Thanks for the explanations. This thread will be useful for many folks i believe.

Best regards

On 2/23/21 4:15 PM, Robert Muir wrote:

On Tue, Feb 23, 2021 at 4:07 PM <ba...@oracle.com>> wrote:

What i want to achieve: Problem statement:

base case is disk based Lucene index with FSDirectory

speedup case was supposed to be in memory Lucene index with MMapDirectory

On 64-bit systems, FSDirectory just invokes MMapDirectory already. So you don't need to do anything.

Either way MMapDirectory or NIOFSDirectory are doing the same thing: reading your index as a normal file and letting the operating system cache it.
The MMapDirectory is just better because it avoids some overheads, such as read() system call, copying and buffering into java memory space, etc etc.
Some of these overheads are only getting worse, e.g. spectre/meltdown-type fixes make syscalls 8x slower on my computer. So it is good that MMapDirectory avoids it.

So I suggest just stop fighting the operating system, don't give your J2EE container huge amounts of ram, let the kernel do its job.
If you want to "warm" a cold system because nothing is in kernel's cache, then look into preload and so on. It is just "reading files" to get them cached.