You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by "kent.fitch" <ke...@projectcomputing.com> on 2006/03/13 03:57:44 UTC
Good MMapDirectory performance
I thought I'd post some good news about MMapDirectory as
the comments in the release notes are quite downbeat about
its performance. In some environments MMapDirectory
provides a big improvement.
Our test application is an index of 11.4 million
documents which are derived from MARC (bibliographic)
catalogue records. Our aim is to build a system
to demonstrate relevance ranking and result clustering
for library union catalogue searching (a "union"
catalogue accumulates/merges records from multiple
ibraries).
Our main index component sizes:
fdt 17GB
fdx 91MB
tis 82MB
frq 45MB
prx 11MB
tii 1.2 MB
We have a separate Lucence index (not discussed further)
which stores the MARC records.
Each document has many fields. We'll probably reduce the
number after we decide on the best search strategies, but
lots of fields gives us lots of flexability whilst testing
search and ranking strategies.
Stored and unindexed fields, used for summary results:
display title
display author
display publication details
holdingsCount (number of libraries holding)
Tokenized indices:
title
author
subject
genre
keyword (all text)
Keyword (untokenized) indices:
title
author
subject
genre
audience
Dewey/LC classification
language
isbn/issn
publication date (date range code)
unique bibliographic id
"Wildcard" Tokenized indices created by a custom "stub"
analyzer which reduces a term to its first few characters:
title
author
subject
keyword
Field boosts are set for some fields. For example, "title"
"sub title", "series title", "component title" are all
stored as "title" but with different field boosts (as a
match on normal title is deemed more relevant than a match
on series title).
The document boost is set to the sqrt of the holdingsCount
(favouring "popular" resources).
The user interface supports searching and refining searches
on specific fields but the most common search is created
from a single google style search box. Here's a typical
query generated from a 2 word search:
+(titleWords:"franz kafka^4.0"
authorWords:"franz kafka^3.0"
subjectWords:"franz kafka^3.0"
keywords:"franz kafka^1.4"
title:franz kafka^4.0
(+titleWords:franz +titleWords:kafka^3.0)
author:franz kafka^3.0
+authorWords:franz +authorWords:kafka^2.0)
subject:franz kafka^3.0
(+subjectWords:franz +subjectWords:kafka^1.5)
(+genreWords:franz +genreWords:kafka^2.0)
(+keywords:franz +keywords:kafka)
(+titleWildcard:fra +titleWildcard:kaf^0.7)
(+authorWildcard:fra +authorWildcard:kaf^0.7)
(+subjectWildcard:fra +subjectWildcard:kaf^0.7)
(+keywordWildcard:fra +keywordWildcard:kaf^0.2)
)
It generated 1635 hits. We then read the first 700
documents in the hit list and extract the date, subject,
author, genre, Dewey/LC classification and audience
fields for each, accumulating the popularity of each.
Using this data, for each of the subject, author, genre,
Dewey/LC and audience categories, we find the 30 most
popular field values and for each of these we query the
index to find their frequency in the entire index.
We then render the first 100 document results (title,
author, publication details, holdings) and the top 30
for each of subject, author, genre, Dewey/KC and audience,
ordering each list by the popularity of the term in the
hit results (sample of the first 700) and rendering the
size of the text based on the frequency of the term in
the entire database (a bit like the Flickr tag popularity
lists). We also render a graph of hit results by date
range.
The initial search is very quick - typically a small
number of tens of millsecs. The "clustering" takes
much longer - reading up to 700 records, extracting
all those fields, sorting to get the top 30 of each
field category, looking up the frequency of each term
in the database.
The test machine was a SunFire440 with 2 x 1.593GHz
UltraSPARC-IIIi processors and 8GB of memory running
Solaris 9, Java 1.5 in 64 bit mode, Jetty. The Lucene data
directory is stored on a local 10K SCSI disk.
The benchmark consisted of running 13,142 representative
and unique search phrases collected from another system.
The search phrases are unsorted. The client (testing)
system is run on another unloaded computer and was
configured to run a varying number of threads representing
different loads. The results discussed here were
produced with 3 threads - 3 simultaneous requests
and the response rates are as seen by the client at
the completion of the request when the last byte of
the response is received.
The Jetty/Lucene JVM was run with:
-ms1200M -mx1200M -d64 -server -verbose:gc
The JVM (and file cache!) were "warmed up" with
a first pass before results were recorded.
The first test was run with Lucene 1.4.1 and achieved
3.3 responses/sec. (The time to generate the response
includes the Jetty "client" time to process and render
the result). CPU Utilisation was low (typically <= 40%),
but IO rates were not very high. We mirrored the
disk and achieved 3.7 responses/sec, but still CPU
utilisation rarely went over 50%.
We moved to Lucene 1.5.1 and with the same configuration
(mirrored) achieved 3.6 responses/sec.
We then set the parameter:
-Dorg.apache.lucene.FSDirectory.class=org.apache.lucene.store.MMapDirectory
to use MMapDirectory and achieved 8.1 responses/sec
with very high CPU utilisation (over 90%). Running
a separate (unseen) set of 10,743 search terms without
a "warm up" achieved 7.8 responses/sec.
With 3 simultaneous requests the total response time
profile as recorded at the client was:
< 200ms for 40% of requests
< 500ms for 80% of requests
< 800ms for 90% of requests.
I read from Peter Keegan's recent postings:
"The Lucene server is using MMapDirectory. I'm running
the jvm with -Xmx16000M. Peak memory usage of the jvm
on Linux is about 6GB and 7.8GB on windows."
We don't have nearly as much memory as Peter but I
wonder whether he is gaining anything with such
a large heap. The file buffers allocated via MMap
reside outside the JVM heap. We notice that although
our JVM heap is 1.2GB max (and regularly reduces
to ~400MB after GC), the process expands to use all
available memory with MMap, and a big Java heap
means less memory for MMap to use(?).
We are very happy with Lucene/MMAP performance given the
extensive processing undertaken for the result clustering.
Kent Fitch
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Good MMapDirectory performance
Posted by Peter Keegan <pe...@gmail.com>.
- I read from Peter Keegan's recent postings:
- "The Lucene server is using MMapDirectory. I'm running
- the jvm with -Xmx16000M. Peak memory usage of the jvm
- on Linux is about 6GB and 7.8GB on windows."
- We don't have nearly as much memory as Peter but I
- wonder whether he is gaining anything with such
- a large heap.
My application gets better throughput with more VM, but that is probably due
to heavy use of ByteBuffers in the application, not VM for Lucene.
Peter
On 3/12/06, kent.fitch <ke...@projectcomputing.com> wrote:
>
> I thought I'd post some good news about MMapDirectory as
> the comments in the release notes are quite downbeat about
> its performance. In some environments MMapDirectory
> provides a big improvement.
>
> Our test application is an index of 11.4 million
> documents which are derived from MARC (bibliographic)
> catalogue records. Our aim is to build a system
> to demonstrate relevance ranking and result clustering
> for library union catalogue searching (a "union"
> catalogue accumulates/merges records from multiple
> ibraries).
>
> Our main index component sizes:
> fdt 17GB
> fdx 91MB
> tis 82MB
> frq 45MB
> prx 11MB
> tii 1.2 MB
>
> We have a separate Lucence index (not discussed further)
> which stores the MARC records.
>
> Each document has many fields. We'll probably reduce the
> number after we decide on the best search strategies, but
> lots of fields gives us lots of flexability whilst testing
> search and ranking strategies.
>
> Stored and unindexed fields, used for summary results:
> display title
> display author
> display publication details
> holdingsCount (number of libraries holding)
>
> Tokenized indices:
> title
> author
> subject
> genre
> keyword (all text)
>
> Keyword (untokenized) indices:
> title
> author
> subject
> genre
> audience
> Dewey/LC classification
> language
> isbn/issn
> publication date (date range code)
> unique bibliographic id
>
> "Wildcard" Tokenized indices created by a custom "stub"
> analyzer which reduces a term to its first few characters:
> title
> author
> subject
> keyword
>
> Field boosts are set for some fields. For example, "title"
> "sub title", "series title", "component title" are all
> stored as "title" but with different field boosts (as a
> match on normal title is deemed more relevant than a match
> on series title).
>
> The document boost is set to the sqrt of the holdingsCount
> (favouring "popular" resources).
>
> The user interface supports searching and refining searches
> on specific fields but the most common search is created
> from a single google style search box. Here's a typical
> query generated from a 2 word search:
>
> +(titleWords:"franz kafka^4.0"
> authorWords:"franz kafka^3.0"
> subjectWords:"franz kafka^3.0"
> keywords:"franz kafka^1.4"
> title:franz kafka^4.0
> (+titleWords:franz +titleWords:kafka^3.0)
> author:franz kafka^3.0
> +authorWords:franz +authorWords:kafka^2.0)
> subject:franz kafka^3.0
> (+subjectWords:franz +subjectWords:kafka^1.5)
> (+genreWords:franz +genreWords:kafka^2.0)
> (+keywords:franz +keywords:kafka)
> (+titleWildcard:fra +titleWildcard:kaf^0.7)
> (+authorWildcard:fra +authorWildcard:kaf^0.7)
> (+subjectWildcard:fra +subjectWildcard:kaf^0.7)
> (+keywordWildcard:fra +keywordWildcard:kaf^0.2)
> )
>
> It generated 1635 hits. We then read the first 700
> documents in the hit list and extract the date, subject,
> author, genre, Dewey/LC classification and audience
> fields for each, accumulating the popularity of each.
>
> Using this data, for each of the subject, author, genre,
> Dewey/LC and audience categories, we find the 30 most
> popular field values and for each of these we query the
> index to find their frequency in the entire index.
>
> We then render the first 100 document results (title,
> author, publication details, holdings) and the top 30
> for each of subject, author, genre, Dewey/KC and audience,
> ordering each list by the popularity of the term in the
> hit results (sample of the first 700) and rendering the
> size of the text based on the frequency of the term in
> the entire database (a bit like the Flickr tag popularity
> lists). We also render a graph of hit results by date
> range.
>
> The initial search is very quick - typically a small
> number of tens of millsecs. The "clustering" takes
> much longer - reading up to 700 records, extracting
> all those fields, sorting to get the top 30 of each
> field category, looking up the frequency of each term
> in the database.
>
> The test machine was a SunFire440 with 2 x 1.593GHz
> UltraSPARC-IIIi processors and 8GB of memory running
> Solaris 9, Java 1.5 in 64 bit mode, Jetty. The Lucene data
> directory is stored on a local 10K SCSI disk.
>
> The benchmark consisted of running 13,142 representative
> and unique search phrases collected from another system.
> The search phrases are unsorted. The client (testing)
> system is run on another unloaded computer and was
> configured to run a varying number of threads representing
> different loads. The results discussed here were
> produced with 3 threads - 3 simultaneous requests
> and the response rates are as seen by the client at
> the completion of the request when the last byte of
> the response is received.
>
> The Jetty/Lucene JVM was run with:
> -ms1200M -mx1200M -d64 -server -verbose:gc
> The JVM (and file cache!) were "warmed up" with
> a first pass before results were recorded.
>
> The first test was run with Lucene 1.4.1 and achieved
> 3.3 responses/sec. (The time to generate the response
> includes the Jetty "client" time to process and render
> the result). CPU Utilisation was low (typically <= 40%),
> but IO rates were not very high. We mirrored the
> disk and achieved 3.7 responses/sec, but still CPU
> utilisation rarely went over 50%.
>
> We moved to Lucene 1.5.1 and with the same configuration
> (mirrored) achieved 3.6 responses/sec.
>
> We then set the parameter:
> -
> Dorg.apache.lucene.FSDirectory.class=org.apache.lucene.store.MMapDirectory
> to use MMapDirectory and achieved 8.1 responses/sec
> with very high CPU utilisation (over 90%). Running
> a separate (unseen) set of 10,743 search terms without
> a "warm up" achieved 7.8 responses/sec.
>
> With 3 simultaneous requests the total response time
> profile as recorded at the client was:
>
> < 200ms for 40% of requests
> < 500ms for 80% of requests
> < 800ms for 90% of requests.
>
> I read from Peter Keegan's recent postings:
> "The Lucene server is using MMapDirectory. I'm running
> the jvm with -Xmx16000M. Peak memory usage of the jvm
> on Linux is about 6GB and 7.8GB on windows."
>
> We don't have nearly as much memory as Peter but I
> wonder whether he is gaining anything with such
> a large heap. The file buffers allocated via MMap
> reside outside the JVM heap. We notice that although
> our JVM heap is 1.2GB max (and regularly reduces
> to ~400MB after GC), the process expands to use all
> available memory with MMap, and a big Java heap
> means less memory for MMap to use(?).
>
> We are very happy with Lucene/MMAP performance given the
> extensive processing undertaken for the result clustering.
>
> Kent Fitch
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>