You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Li Li <fa...@gmail.com> on 2014/12/03 03:46:11 UTC
how to "load" mmap directory into memory?
I am using mmap fs directory in lucene. My index is small (about 3GB
in disk) and I have plenty of memory available. The problem is that
when the term is first queried, it's slow. How can I "load" all
directory into memory? One solution is using many query to "warm" it
up. But I can't query all terms and maybe some terms are still not in
memory. Any other methods?
If I write a script(or java program) to read the index file, will it
load all into memory if I have enough memory?
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: how to "load" mmap directory into memory?
Posted by br...@gmail.com.
Canuaddalotofnonsenseorshallwebedirect!?!?! X
Sent from my BlackBerry® wireless device
-----Original Message-----
From: "Uwe Schindler" <uw...@thetaphi.de>
Date: Wed, 3 Dec 2014 11:54:27
To: <ja...@lucene.apache.org>
Reply-To: java-user@lucene.apache.org
Subject: RE: how to "load" mmap directory into memory?
Hi,
the first searches are always slower, which is not only caused by the FS cache. In general, searchers must be warmed, so you execute a set of search queries or calculate facets initially after opening (or reopening) an IndexReader. This will have the side effect of also warming the disk cache. But it also initializes all required structures inside Lucene. So instead of forcefully loading the whole index files into fs-cache, its better to use IndexReaderWarmer(http://lucene.apache.org/core/4_10_2/core/org/apache/lucene/index/IndexWriter.IndexReaderWarmer.html), which can be used with SearcherManager in Lucene to warm searchers.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Li Li [mailto:fancyerii@gmail.com]
> Sent: Wednesday, December 03, 2014 3:46 AM
> To: java-user@lucene.apache.org
> Subject: how to "load" mmap directory into memory?
>
> I am using mmap fs directory in lucene. My index is small (about 3GB in disk)
> and I have plenty of memory available. The problem is that when the term is
> first queried, it's slow. How can I "load" all directory into memory? One
> solution is using many query to "warm" it up. But I can't query all terms and
> maybe some terms are still not in memory. Any other methods?
> If I write a script(or java program) to read the index file, will it load all into
> memory if I have enough memory?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: how to "load" mmap directory into memory?
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
the first searches are always slower, which is not only caused by the FS cache. In general, searchers must be warmed, so you execute a set of search queries or calculate facets initially after opening (or reopening) an IndexReader. This will have the side effect of also warming the disk cache. But it also initializes all required structures inside Lucene. So instead of forcefully loading the whole index files into fs-cache, its better to use IndexReaderWarmer(http://lucene.apache.org/core/4_10_2/core/org/apache/lucene/index/IndexWriter.IndexReaderWarmer.html), which can be used with SearcherManager in Lucene to warm searchers.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Li Li [mailto:fancyerii@gmail.com]
> Sent: Wednesday, December 03, 2014 3:46 AM
> To: java-user@lucene.apache.org
> Subject: how to "load" mmap directory into memory?
>
> I am using mmap fs directory in lucene. My index is small (about 3GB in disk)
> and I have plenty of memory available. The problem is that when the term is
> first queried, it's slow. How can I "load" all directory into memory? One
> solution is using many query to "warm" it up. But I can't query all terms and
> maybe some terms are still not in memory. Any other methods?
> If I write a script(or java program) to read the index file, will it load all into
> memory if I have enough memory?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: how to "load" mmap directory into memory?
Posted by br...@gmail.com.
"Indin _bull?
Sent from my BlackBerry® wireless device
-----Original Message-----
From: Toke Eskildsen <te...@statsbiblioteket.dk>
Date: Wed, 3 Dec 2014 08:25:04
To: java-user@lucene.apache.org<ja...@lucene.apache.org>
Reply-To: java-user@lucene.apache.org
Subject: RE: how to "load" mmap directory into memory?
Li Li [fancyerii@gmail.com] wrote:
> I am using mmap fs directory in lucene. My index is small (about 3GB
> in disk) and I have plenty of memory available. The problem is that
> when the term is first queried, it's slow. How can I "load" all
> directory into memory?
Yes. One of the smart things about MMap is that it is shared with the disk cache, so as long as you read the bytes somehow, for example with the Java program you suggest, it will be available in memory for Lucene.
On linux, the simplest is to 'cat myindexfolder/* > /dev/null'. Do not use 'cp' as it is smart enough to bypass the operation if the destination is /dev/null.
Caveat: This does not guarantee that your index stays fully cached. It can be evicted just like all other disk cache, if other programs perform enough disk operations.
- Toke Eskildsen
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: how to "load" mmap directory into memory?
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
Li Li [fancyerii@gmail.com] wrote:
> I am using mmap fs directory in lucene. My index is small (about 3GB
> in disk) and I have plenty of memory available. The problem is that
> when the term is first queried, it's slow. How can I "load" all
> directory into memory?
Yes. One of the smart things about MMap is that it is shared with the disk cache, so as long as you read the bytes somehow, for example with the Java program you suggest, it will be available in memory for Lucene.
On linux, the simplest is to 'cat myindexfolder/* > /dev/null'. Do not use 'cp' as it is smart enough to bypass the operation if the destination is /dev/null.
Caveat: This does not guarantee that your index stays fully cached. It can be evicted just like all other disk cache, if other programs perform enough disk operations.
- Toke Eskildsen
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org