You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Mark Florence <mf...@airsmail.com> on 2004/06/28 22:51:11 UTC
Running OutOfMemory while optimizing and searching
Hi, I'm using Lucene to index ~3.5M documents, over about 50 fields. The Lucene
index itself is ~10.5GB, spread over ~7,000 files. Some of these files are
"large" -- that is, several PRX files are ~1.5GB.
Lucene runs on a dedicated server (Linux on a 1Ghz Dell, with 1GB RAM). Clients
on other machines use RMI to perform reads / writes. Each night the server
automatically performs an optimize.
The problem is that the optimize now dies with an OutOfMemory exception, even
when the JVM heap size is set to its maximum of 2GB. I need to optimize, because
as the number of Lucene files grows, search performance becomes unacceptable.
Search performance is also adversely affected because I've had to effectively
single-thread reads and writes. I was using a simple read / write lock
mechanism, allowing multiple readers to simultaneously search, but now more than
3-4 simultaneous readers will also cause an OutOfMemory condition. Searches can
take as long as 30-40 seconds, and with single-threading, that's crippling the
main client application.
Needless to say, the Lucene index is mission-critical, and must run 24/7.
I've seen other posts along this same vein, but no definite consensus. Is my
problem simply inadequate hardware? Should I run on a 64-bit platform, where I
can allocate a Java heap of > 2GB?
Or could there be something fundamentally "wrong" with my index? I should add
that I've just spent about a week (!!) rebuilding from scratch, over all 3.5M
documents.
-- Many thanks for any help! Mark Florence
Re: Optimizing Collections
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Don,
Use IndexWriter's optimize() method:
IndexWriter writer = new IndexWriter(.....)
writer.optimize();
writer.close();
Otis
--- Don Vaillancourt <do...@webimpact.com> wrote:
> Hello All,
>
> I've finished with the Lucene collections. I'm building collections
> based
> on PDFs and database queries and everything is winderful.
>
> I remember reading somewhere that you could optimize a Lucene
> collection. But I can't remember where and cannot find anything on
> that. I've also read it here as well.
>
> Could someone point me to the right direction on how to optimize a
> collection?
>
>
> Thanks
>
>
> Don Vaillancourt
> Director of Software Development
>
> WEB IMPACT INC.
> 416-815-2000 ext. 245
> email: donv@web-impact.com
> web: http://www.web-impact.com
>
>
>
>
> This email message is intended only for the addressee(s)
> and contains information that may be confidential and/or
> copyright. If you are not the intended recipient please
> notify the sender by reply email and immediately delete
> this email. Use, disclosure or reproduction of this email
> by anyone other than the intended recipient(s) is strictly
> prohibited. No representation is made that this email or
> any attachments are free of viruses. Virus scanning is
> recommended and is the responsibility of the recipient.
>
>
>
>
>
>
>
>
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Optimizing Collections
Posted by Don Vaillancourt <do...@webimpact.com>.
Hello All,
I've finished with the Lucene collections. I'm building collections based
on PDFs and database queries and everything is winderful.
I remember reading somewhere that you could optimize a Lucene
collection. But I can't remember where and cannot find anything on
that. I've also read it here as well.
Could someone point me to the right direction on how to optimize a collection?
Thanks
Don Vaillancourt
Director of Software Development
WEB IMPACT INC.
416-815-2000 ext. 245
email: donv@web-impact.com
web: http://www.web-impact.com
This email message is intended only for the addressee(s)
and contains information that may be confidential and/or
copyright. If you are not the intended recipient please
notify the sender by reply email and immediately delete
this email. Use, disclosure or reproduction of this email
by anyone other than the intended recipient(s) is strictly
prohibited. No representation is made that this email or
any attachments are free of viruses. Virus scanning is
recommended and is the responsibility of the recipient.
RE: Running OutOfMemory while optimizing and searching
Posted by Mark Florence <mf...@airsmail.com>.
Thanks, Jim, your arithmetic is compelling. The queries
are ad hoc, of course, but an average query might easily
search on 10-15 fields. We avoid wildcards except in a few
special cases, because they can be very slow. Example: to
search for first name beginning with M -- firstName:M* --
takes well over 40 seconds on my index.
Given these numbers, what is your experience of hardware
necessary to get decent performance out of large, highly
interactive Lucene indexes? I'm thinking of Alex McManus's
recent post "Making a case for Lucene" where he posits an
index over 100M documents, a place where we plan to be too.
It seems that all my issues are amenable to a hardware
solution. I guess I just don't know enough to make a
recommendation to my people as to exactly what kind of
iron we need :(
-- Mark
-----Original Message-----
From: James Dunn [mailto:james_h_dunn@yahoo.com]
Sent: Tuesday, June 29, 2004 07:29 pm
To: Lucene Users List
Subject: RE: Running OutOfMemory while optimizing and searching
Mark,
What do your queries look like? The memory required
for a query can be computed by the following equation:
1 Byte * Number of fields in your query * Number of
docs in your index
So if your query searches on all 50 fields of your 3.5
Million document index then each search would take
about 175MB. If your 3-4 searches run concurrently
then that's about 525MB to 700MB chewed up at once.
Also, if your queries use wildcards, the memory
requirements could be much greater.
Hope that helps,
Jim
--- Mark Florence <mf...@airsmail.com> wrote:
> Otis, Thanks for considering this problem.
>
> I'm using all the default parameters -- and still
> scratching my head!
>
> I can't see any that would control memory usage.
> Plus, a 2GB heap is
> quite big. I see others have indexes bigger than
> mine, so I'm not sure
> how to find out why mine is throwing OutOfMemory --
> not only on the
> optimize, but when 3-4 searchers are running, too.
>
> -- Mark
>
> -----Original Message-----
> From: Otis Gospodnetic
> [mailto:otis_gospodnetic@yahoo.com]
> Sent: Tuesday, June 29, 2004 01:02 am
> To: Lucene Users List
> Subject: Re: Running OutOfMemory while optimizing
> and searching
>
>
> Mark,
>
> Tough situation. I hate when things like this
> happen on production :(.
> You are not mentioning what you are using for
> various IndexWriter
> parameters. You may be able to get this working by
> tweaking them (see
>
http://jakarta.apache.org/lucene/docs/api/org/apache/lucene/index/IndexWrite
> r.html#field_summary).
> Hm, now that I think about it, I am not sure if
> those are considered
> during index optimization. I'll try checking the
> sources later.
>
> Otis
>
> --- Mark Florence <mf...@airsmail.com> wrote:
> > Hi, I'm using Lucene to index ~3.5M documents,
> over about 50 fields.
> > The Lucene
> > index itself is ~10.5GB, spread over ~7,000 files.
> Some of these
> > files are
> > "large" -- that is, several PRX files are ~1.5GB.
> >
> > Lucene runs on a dedicated server (Linux on a 1Ghz
> Dell, with 1GB
> > RAM). Clients
> > on other machines use RMI to perform reads /
> writes. Each night the
> > server
> > automatically performs an optimize.
> >
> > The problem is that the optimize now dies with an
> OutOfMemory
> > exception, even
> > when the JVM heap size is set to its maximum of
> 2GB. I need to
> > optimize, because
> > as the number of Lucene files grows, search
> performance becomes
> > unacceptable.
> >
> > Search performance is also adversely affected
> because I've had to
> > effectively
> > single-thread reads and writes. I was using a
> simple read / write
> > lock
> > mechanism, allowing multiple readers to
> simultaneously search, but
> > now more than
> > 3-4 simultaneous readers will also cause an
> OutOfMemory condition.
> > Searches can
> > take as long as 30-40 seconds, and with
> single-threading, that's
> > crippling the
> > main client application.
> >
> > Needless to say, the Lucene index is
> mission-critical, and must run
> > 24/7.
> >
> > I've seen other posts along this same vein, but no
> definite
> > consensus. Is my
> > problem simply inadequate hardware? Should I run
> on a 64-bit
> > platform, where I
> > can allocate a Java heap of > 2GB?
> >
> > Or could there be something fundamentally "wrong"
> with my index? I
> > should add
> > that I've just spent about a week (!!) rebuilding
> from scratch, over
> > all 3.5M
> > documents.
> >
> > -- Many thanks for any help! Mark Florence
> >
> >
> >
>
>
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
>
>
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
RE: Running OutOfMemory while optimizing and searching
Posted by Paul Smith <ps...@aconex.com>.
Wow, I have say that those sort of numbers are concerning to me... Now I
know 3.5 million documents is a lot, but still... What would be causing a
query to require and hold that much memory? I could understand that it
surely would be doing a lot of memory work, but why would it need to hold
onto/grab that much memory for the length of the query? (this is getting
into the internals a bit, but it's always good to know what's going on under
the hood from a design decision point of view and how I would have to
structure an App to handle this sort of load).
Cheers,
Paul Smith
> 1 Byte * Number of fields in your query * Number of
> docs in your index
>
> So if your query searches on all 50 fields of your 3.5
> Million document index then each search would take
> about 175MB. If your 3-4 searches run concurrently
> then that's about 525MB to 700MB chewed up at once.
>
> Also, if your queries use wildcards, the memory
> requirements could be much greater.
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
RE: Running OutOfMemory while optimizing and searching
Posted by James Dunn <ja...@yahoo.com>.
Mark,
What do your queries look like? The memory required
for a query can be computed by the following equation:
1 Byte * Number of fields in your query * Number of
docs in your index
So if your query searches on all 50 fields of your 3.5
Million document index then each search would take
about 175MB. If your 3-4 searches run concurrently
then that's about 525MB to 700MB chewed up at once.
Also, if your queries use wildcards, the memory
requirements could be much greater.
Hope that helps,
Jim
--- Mark Florence <mf...@airsmail.com> wrote:
> Otis, Thanks for considering this problem.
>
> I'm using all the default parameters -- and still
> scratching my head!
>
> I can't see any that would control memory usage.
> Plus, a 2GB heap is
> quite big. I see others have indexes bigger than
> mine, so I'm not sure
> how to find out why mine is throwing OutOfMemory --
> not only on the
> optimize, but when 3-4 searchers are running, too.
>
> -- Mark
>
> -----Original Message-----
> From: Otis Gospodnetic
> [mailto:otis_gospodnetic@yahoo.com]
> Sent: Tuesday, June 29, 2004 01:02 am
> To: Lucene Users List
> Subject: Re: Running OutOfMemory while optimizing
> and searching
>
>
> Mark,
>
> Tough situation. I hate when things like this
> happen on production :(.
> You are not mentioning what you are using for
> various IndexWriter
> parameters. You may be able to get this working by
> tweaking them (see
>
http://jakarta.apache.org/lucene/docs/api/org/apache/lucene/index/IndexWrite
> r.html#field_summary).
> Hm, now that I think about it, I am not sure if
> those are considered
> during index optimization. I'll try checking the
> sources later.
>
> Otis
>
> --- Mark Florence <mf...@airsmail.com> wrote:
> > Hi, I'm using Lucene to index ~3.5M documents,
> over about 50 fields.
> > The Lucene
> > index itself is ~10.5GB, spread over ~7,000 files.
> Some of these
> > files are
> > "large" -- that is, several PRX files are ~1.5GB.
> >
> > Lucene runs on a dedicated server (Linux on a 1Ghz
> Dell, with 1GB
> > RAM). Clients
> > on other machines use RMI to perform reads /
> writes. Each night the
> > server
> > automatically performs an optimize.
> >
> > The problem is that the optimize now dies with an
> OutOfMemory
> > exception, even
> > when the JVM heap size is set to its maximum of
> 2GB. I need to
> > optimize, because
> > as the number of Lucene files grows, search
> performance becomes
> > unacceptable.
> >
> > Search performance is also adversely affected
> because I've had to
> > effectively
> > single-thread reads and writes. I was using a
> simple read / write
> > lock
> > mechanism, allowing multiple readers to
> simultaneously search, but
> > now more than
> > 3-4 simultaneous readers will also cause an
> OutOfMemory condition.
> > Searches can
> > take as long as 30-40 seconds, and with
> single-threading, that's
> > crippling the
> > main client application.
> >
> > Needless to say, the Lucene index is
> mission-critical, and must run
> > 24/7.
> >
> > I've seen other posts along this same vein, but no
> definite
> > consensus. Is my
> > problem simply inadequate hardware? Should I run
> on a 64-bit
> > platform, where I
> > can allocate a Java heap of > 2GB?
> >
> > Or could there be something fundamentally "wrong"
> with my index? I
> > should add
> > that I've just spent about a week (!!) rebuilding
> from scratch, over
> > all 3.5M
> > documents.
> >
> > -- Many thanks for any help! Mark Florence
> >
> >
> >
>
>
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
>
>
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
RE: Running OutOfMemory while optimizing and searching
Posted by Mark Florence <mf...@airsmail.com>.
Otis, Thanks for considering this problem.
I'm using all the default parameters -- and still scratching my head!
I can't see any that would control memory usage. Plus, a 2GB heap is
quite big. I see others have indexes bigger than mine, so I'm not sure
how to find out why mine is throwing OutOfMemory -- not only on the
optimize, but when 3-4 searchers are running, too.
-- Mark
-----Original Message-----
From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
Sent: Tuesday, June 29, 2004 01:02 am
To: Lucene Users List
Subject: Re: Running OutOfMemory while optimizing and searching
Mark,
Tough situation. I hate when things like this happen on production :(.
You are not mentioning what you are using for various IndexWriter
parameters. You may be able to get this working by tweaking them (see
http://jakarta.apache.org/lucene/docs/api/org/apache/lucene/index/IndexWrite
r.html#field_summary).
Hm, now that I think about it, I am not sure if those are considered
during index optimization. I'll try checking the sources later.
Otis
--- Mark Florence <mf...@airsmail.com> wrote:
> Hi, I'm using Lucene to index ~3.5M documents, over about 50 fields.
> The Lucene
> index itself is ~10.5GB, spread over ~7,000 files. Some of these
> files are
> "large" -- that is, several PRX files are ~1.5GB.
>
> Lucene runs on a dedicated server (Linux on a 1Ghz Dell, with 1GB
> RAM). Clients
> on other machines use RMI to perform reads / writes. Each night the
> server
> automatically performs an optimize.
>
> The problem is that the optimize now dies with an OutOfMemory
> exception, even
> when the JVM heap size is set to its maximum of 2GB. I need to
> optimize, because
> as the number of Lucene files grows, search performance becomes
> unacceptable.
>
> Search performance is also adversely affected because I've had to
> effectively
> single-thread reads and writes. I was using a simple read / write
> lock
> mechanism, allowing multiple readers to simultaneously search, but
> now more than
> 3-4 simultaneous readers will also cause an OutOfMemory condition.
> Searches can
> take as long as 30-40 seconds, and with single-threading, that's
> crippling the
> main client application.
>
> Needless to say, the Lucene index is mission-critical, and must run
> 24/7.
>
> I've seen other posts along this same vein, but no definite
> consensus. Is my
> problem simply inadequate hardware? Should I run on a 64-bit
> platform, where I
> can allocate a Java heap of > 2GB?
>
> Or could there be something fundamentally "wrong" with my index? I
> should add
> that I've just spent about a week (!!) rebuilding from scratch, over
> all 3.5M
> documents.
>
> -- Many thanks for any help! Mark Florence
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Running OutOfMemory while optimizing and searching
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Mark,
Tough situation. I hate when things like this happen on production :(.
You are not mentioning what you are using for various IndexWriter
parameters. You may be able to get this working by tweaking them (see
http://jakarta.apache.org/lucene/docs/api/org/apache/lucene/index/IndexWriter.html#field_summary).
Hm, now that I think about it, I am not sure if those are considered
during index optimization. I'll try checking the sources later.
Otis
--- Mark Florence <mf...@airsmail.com> wrote:
> Hi, I'm using Lucene to index ~3.5M documents, over about 50 fields.
> The Lucene
> index itself is ~10.5GB, spread over ~7,000 files. Some of these
> files are
> "large" -- that is, several PRX files are ~1.5GB.
>
> Lucene runs on a dedicated server (Linux on a 1Ghz Dell, with 1GB
> RAM). Clients
> on other machines use RMI to perform reads / writes. Each night the
> server
> automatically performs an optimize.
>
> The problem is that the optimize now dies with an OutOfMemory
> exception, even
> when the JVM heap size is set to its maximum of 2GB. I need to
> optimize, because
> as the number of Lucene files grows, search performance becomes
> unacceptable.
>
> Search performance is also adversely affected because I've had to
> effectively
> single-thread reads and writes. I was using a simple read / write
> lock
> mechanism, allowing multiple readers to simultaneously search, but
> now more than
> 3-4 simultaneous readers will also cause an OutOfMemory condition.
> Searches can
> take as long as 30-40 seconds, and with single-threading, that's
> crippling the
> main client application.
>
> Needless to say, the Lucene index is mission-critical, and must run
> 24/7.
>
> I've seen other posts along this same vein, but no definite
> consensus. Is my
> problem simply inadequate hardware? Should I run on a 64-bit
> platform, where I
> can allocate a Java heap of > 2GB?
>
> Or could there be something fundamentally "wrong" with my index? I
> should add
> that I've just spent about a week (!!) rebuilding from scratch, over
> all 3.5M
> documents.
>
> -- Many thanks for any help! Mark Florence
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org