You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Nadav Har'El <NY...@il.ibm.com> on 2006/06/12 14:13:10 UTC
Does more memory help Lucene?
Hi,
I am trying to index a huge collection of documents - several hundreds of
gigabytes.
Needless to say, I'm trying to squeeze every ounce of performance from my
machine, to get this indexing done in a sensible amount of time.
Making use of the fact that my machine has two CPUs was easy: I just run
two indexing threads, creating two separate indices which can later be
merged, or searched with a MultiSearch.
What I couldn't figure out how to use, however, was the abundant memory (2
GB) that this machine has.
I tried playing with IndexWriter.setMaxBufferedDocs(), and noticed that
there is no speed gain after I set it to 1000, at which point the running
Lucene takes up just 70 MB of memory, or 140 MB for the two threads.
Is there a way for Lucene to make use of the much larger memory I have, to
speed up the indexing process? Does having a huge memory somehow improve
the speed of huge merges, for example?
Thanks,
Nadav.
--
Nadav Har'El
nyh@il.ibm.com
+972-4-829-6326
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Does more memory help Lucene?
Posted by Nadav Har'El <NY...@il.ibm.com>.
"Michael D. Curtin" <mi...@curtin.com> wrote on 12/06/2006 03:49:53 PM:
> Nadav Har'El wrote:
>
> > What I couldn't figure out how to use, however, was the abundant memory
(2
> > GB) that this machine has.
> >
> > I tried playing with IndexWriter.setMaxBufferedDocs(), and noticed that
> > there is no speed gain after I set it to 1000, at which point the
running
> > Lucene takes up just 70 MB of memory, or 140 MB for the two threads.
>
> It may not be a Lucene limit per se, but a JVM limit instead. What are
you
> using for the JVM's heap (via -Xms and -Xmx switches)? For example, I
often
> run with java -Xmx1000m to let the heap grow to a gigabyte, if necessary.
Sure, I used -Xmx500m, for example, but Java still only grew to 70 MB.
When I continued to increase setMaxBufferedDocs(), Java's memory use grew,
but performance did not continue to improve (it even slightly dropped, I
don't know why). So apparently, the problem isn't Java's memory limit,
it's Lucene simply not wishing to use more memory.
> Might I also suggest that you not try to index all of this data in a
single
> invocation of a Java program. That is, index a portion, say 10GB at a
time,
> and then use AddIndexes() later to bring them together. Set your
granularity
> based on the amount of time you can stand to do work over, when the
seemingly
> inevitable problems crop up. It sure would stink to be working on the
1000th
> GB of input data and have a power supply go out, and then have to start
all
> the way over from the beginning! Other checkpointing schemes are
> possible, if
> you have the time and inclination to be more clever, too ...
Good idea.
Thanks,
Nadav.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Does more memory help Lucene?
Posted by "Michael D. Curtin" <mi...@curtin.com>.
Nadav Har'El wrote:
> What I couldn't figure out how to use, however, was the abundant memory (2
> GB) that this machine has.
>
> I tried playing with IndexWriter.setMaxBufferedDocs(), and noticed that
> there is no speed gain after I set it to 1000, at which point the running
> Lucene takes up just 70 MB of memory, or 140 MB for the two threads.
>
> Is there a way for Lucene to make use of the much larger memory I have, to
> speed up the indexing process? Does having a huge memory somehow improve
> the speed of huge merges, for example?
It may not be a Lucene limit per se, but a JVM limit instead. What are you
using for the JVM's heap (via -Xms and -Xmx switches)? For example, I often
run with java -Xmx1000m to let the heap grow to a gigabyte, if necessary.
Might I also suggest that you not try to index all of this data in a single
invocation of a Java program. That is, index a portion, say 10GB at a time,
and then use AddIndexes() later to bring them together. Set your granularity
based on the amount of time you can stand to do work over, when the seemingly
inevitable problems crop up. It sure would stink to be working on the 1000th
GB of input data and have a power supply go out, and then have to start all
the way over from the beginning! Other checkpointing schemes are possible, if
you have the time and inclination to be more clever, too ...
Good luck!
--MDC
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Does more memory help Lucene?
Posted by Dan Armbrust <da...@gmail.com>.
>The reason I'm
> asking this that I'm still trying to figure out whether having a machine
> with huge ram actually helps Lucene, or not.
>
> Thanks,
> Nadav.
>
>
Memory can help a little at index time, but you will mostly be Disk / IO
bound. How fast can you read your data in, how fast can you write it
out. More spinning disks will help more than extra RAM.
At least in my experience.
Dan
--
****************************
Daniel Armbrust
Biomedical Informatics
Mayo Clinic Rochester
daniel.armbrust(at)mayo.edu
http://informatics.mayo.edu/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Does more memory help Lucene?
Posted by Peter Keegan <pe...@gmail.com>.
See my note about overlapping indexing documents with merging:
http://www.gossamer-threads.com/lists/lucene/java-user/34188?search_string=%2Bkeegan%20%2Baddindexes;#34188
Peter
On 6/12/06, Michael D. Curtin <mi...@curtin.com> wrote:
>
> Nadav Har'El wrote:
>
> > Otis Gospodnetic <ot...@yahoo.com> wrote on 12/06/2006
> 04:36:45
> > PM:
> >
> >
> >>Nadav,
> >>
> >>Look up one of my onjava.com Lucene articles, where I talk about
> >>this. You may also want to tell Lucene to merge segments on disk
> >>less frequently, which is what mergeFactor does.
> >
> >
> > Thanks. Can you please point me to the appropriate article (I found one
> > from March 2003, but I'm not sure if it's the one you meant).
> >
> > About mergeFactor() - thanks for the hint, I'll try changing it too (I
> used
> > 20 so far), and see if it helps performance.
> >
> > Still, there is one thing about mergeFactor(), and the merge process,
> that
> > I don't understand: does having more memory help this process at all?
> Does
> > having a large mergeFactor() actually require more memory? The reason
> I'm
> > asking this that I'm still trying to figure out whether having a machine
> > with huge ram actually helps Lucene, or not.
>
> I'm using 1.4.3, so I don't know if things are the same in 2.0. Anyhow, I
> found a significant performance benefit from changing minMergeDocs and
> mergeFactor from their defaults of 10 and 10 to 1,000 and 70,
> respectively.
> The improvement seems to come from a reduction in the number of merges as
> the
> index is created. Each merge involves reading and writing a bunch of data
> already indexed, sometimes everything indexed so far, so it's easy to see
> how
> reducing the number of merges reduces the overall indexing time.
>
> I can't remember why, but I also little benefit to increasing minMergeDocs
> beyond 1000. A lot of time was being spent in the first merge, which
> takes a
> bunch of one-document "segments" in a RAMDirectory and merges them into
> the
> first-level segments on disk. I hacked the code to make this first merge
> (and
> ONLY the first merge) operate on minMergeDocs * mergeFactor documents
> instead,
> which greatly increased the RAM consumption but reduced the indexing
> time. In
> detail, what I started with was:
> a. read minMergeDocs of docs, creating one-doc segments in RAM
> b. read those one-doc RAM segments and merge them
> c. write the merged results to a disk segment
> ...
> i. read mergeFactor first-level disk segments and merge them
> j. write second-level segments to disk
> ...
> p. normal disk-based merging thereafter, as necessary
>
> And what I ended up with was:
> A. read minMergeDocs * mergeFactor docs, and remember them in RAM
> B. write a segment from all the remembered RAM docs (a modified merge)
> ...
> F. normal disk-based merging thereafter, as necessary
>
> In essence, I eliminated that first level merge, one that involved lots
> and
> lots of teeny-weeny I/O operations that were very inefficient.
>
> In my case, steps A & B worked on 70,000 documents instead of 1,000.
> Remembering all those docs required a lot of RAM (almost 2GB), but it
> almost
> tripled indexing performance. Later, I had to knock the 70 down to 35
> (maybe
> because my docs got a lot bigger but I don't remember now), but you get
> the
> idea. I couldn't use a mergeFactor of 70,000 because that's way more file
> descriptors than I could have without recompiling the kernel (I seem to
> remember my limit being 1,024, and each segment took 14 file descriptors).
>
> Hope it helps.
>
> --MDC
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Does more memory help Lucene?
Posted by "Michael D. Curtin" <mi...@curtin.com>.
Nadav Har'El wrote:
> Otis Gospodnetic <ot...@yahoo.com> wrote on 12/06/2006 04:36:45
> PM:
>
>
>>Nadav,
>>
>>Look up one of my onjava.com Lucene articles, where I talk about
>>this. You may also want to tell Lucene to merge segments on disk
>>less frequently, which is what mergeFactor does.
>
>
> Thanks. Can you please point me to the appropriate article (I found one
> from March 2003, but I'm not sure if it's the one you meant).
>
> About mergeFactor() - thanks for the hint, I'll try changing it too (I used
> 20 so far), and see if it helps performance.
>
> Still, there is one thing about mergeFactor(), and the merge process, that
> I don't understand: does having more memory help this process at all? Does
> having a large mergeFactor() actually require more memory? The reason I'm
> asking this that I'm still trying to figure out whether having a machine
> with huge ram actually helps Lucene, or not.
I'm using 1.4.3, so I don't know if things are the same in 2.0. Anyhow, I
found a significant performance benefit from changing minMergeDocs and
mergeFactor from their defaults of 10 and 10 to 1,000 and 70, respectively.
The improvement seems to come from a reduction in the number of merges as the
index is created. Each merge involves reading and writing a bunch of data
already indexed, sometimes everything indexed so far, so it's easy to see how
reducing the number of merges reduces the overall indexing time.
I can't remember why, but I also little benefit to increasing minMergeDocs
beyond 1000. A lot of time was being spent in the first merge, which takes a
bunch of one-document "segments" in a RAMDirectory and merges them into the
first-level segments on disk. I hacked the code to make this first merge (and
ONLY the first merge) operate on minMergeDocs * mergeFactor documents instead,
which greatly increased the RAM consumption but reduced the indexing time. In
detail, what I started with was:
a. read minMergeDocs of docs, creating one-doc segments in RAM
b. read those one-doc RAM segments and merge them
c. write the merged results to a disk segment
...
i. read mergeFactor first-level disk segments and merge them
j. write second-level segments to disk
...
p. normal disk-based merging thereafter, as necessary
And what I ended up with was:
A. read minMergeDocs * mergeFactor docs, and remember them in RAM
B. write a segment from all the remembered RAM docs (a modified merge)
...
F. normal disk-based merging thereafter, as necessary
In essence, I eliminated that first level merge, one that involved lots and
lots of teeny-weeny I/O operations that were very inefficient.
In my case, steps A & B worked on 70,000 documents instead of 1,000.
Remembering all those docs required a lot of RAM (almost 2GB), but it almost
tripled indexing performance. Later, I had to knock the 70 down to 35 (maybe
because my docs got a lot bigger but I don't remember now), but you get the
idea. I couldn't use a mergeFactor of 70,000 because that's way more file
descriptors than I could have without recompiling the kernel (I seem to
remember my limit being 1,024, and each segment took 14 file descriptors).
Hope it helps.
--MDC
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Does more memory help Lucene?
Posted by Nadav Har'El <NY...@il.ibm.com>.
Otis Gospodnetic <ot...@yahoo.com> wrote on 12/06/2006 04:36:45
PM:
> Nadav,
>
> Look up one of my onjava.com Lucene articles, where I talk about
> this. You may also want to tell Lucene to merge segments on disk
> less frequently, which is what mergeFactor does.
Thanks. Can you please point me to the appropriate article (I found one
from March 2003, but I'm not sure if it's the one you meant).
About mergeFactor() - thanks for the hint, I'll try changing it too (I used
20 so far), and see if it helps performance.
Still, there is one thing about mergeFactor(), and the merge process, that
I don't understand: does having more memory help this process at all? Does
having a large mergeFactor() actually require more memory? The reason I'm
asking this that I'm still trying to figure out whether having a machine
with huge ram actually helps Lucene, or not.
Thanks,
Nadav.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Does more memory help Lucene?
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Nadav,
Look up one of my onjava.com Lucene articles, where I talk about this. You may also want to tell Lucene to merge segments on disk less frequently, which is what mergeFactor does.
Otis
----- Original Message ----
From: Nadav Har'El <NY...@il.ibm.com>
To: java-user@lucene.apache.org
Sent: Monday, June 12, 2006 8:13:10 AM
Subject: Does more memory help Lucene?
Hi,
I am trying to index a huge collection of documents - several hundreds of
gigabytes.
Needless to say, I'm trying to squeeze every ounce of performance from my
machine, to get this indexing done in a sensible amount of time.
Making use of the fact that my machine has two CPUs was easy: I just run
two indexing threads, creating two separate indices which can later be
merged, or searched with a MultiSearch.
What I couldn't figure out how to use, however, was the abundant memory (2
GB) that this machine has.
I tried playing with IndexWriter.setMaxBufferedDocs(), and noticed that
there is no speed gain after I set it to 1000, at which point the running
Lucene takes up just 70 MB of memory, or 140 MB for the two threads.
Is there a way for Lucene to make use of the much larger memory I have, to
speed up the indexing process? Does having a huge memory somehow improve
the speed of huge merges, for example?
Thanks,
Nadav.
--
Nadav Har'El
nyh@il.ibm.com
+972-4-829-6326
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org