You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Kurt Vaag <kg...@knowledgeworkers.com> on 2002/02/25 20:15:21 UTC
Build index using RAMDirectory "out of memory" errors
I have been using Lucene for 3 weeks and it rules.
The indexing process can be slow. So I searched the mailgroup archives
and found example code using RAMDirectory to improve indexing speed.
The example code I found was indexing 100,000 files at a time to the
RAMDirectory before writing to disk.
I tried indexing 10,000 files at a time to the RAMDirectory before writing
to disk. This drastically improved indexing times but sometimes I get
"out of memory" errors. I am indexing text files and adding 9 fields from
an Oracle database.
Environment:
Solaris 2.8 with 1G of ram and 2G of swap
Java 1.3.1
Lucene 1.2-rc4
Any ideas for eliminating the "out of memory" errors ?
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Build index using RAMDirectory "out of memory" errors
Posted by Kurt Vaag <kg...@knowledgeworkers.com>.
Thanks Winton,
Thats what it was. I just assumed java would take all the 1G that
it needed. Didn't realize the default was 64M. Also thanks for not
saying RTFM (which I had done but didn't know what TF to do with the
-Xmx option).
-Kurt
-----Original Message-----
From: Winton Davies [mailto:wdavies@cs.stanford.edu]
Sent: Monday, February 25, 2002 12:22 PM
To: Lucene Users List
Subject: Re: Build index using RAMDirectory "out of memory" errors
java -Xmx1000m ....
Sorry if you already tried resizing your heap. Actually with 1.3.1
you could go up above a gig, but really swapping aint gonna help much.
Winton
>I have been using Lucene for 3 weeks and it rules.
>
>The indexing process can be slow. So I searched the mailgroup archives
>and found example code using RAMDirectory to improve indexing speed.
>The example code I found was indexing 100,000 files at a time to the
>RAMDirectory before writing to disk.
>
>I tried indexing 10,000 files at a time to the RAMDirectory before writing
>to disk. This drastically improved indexing times but sometimes I get
>"out of memory" errors. I am indexing text files and adding 9 fields from
>an Oracle database.
>
>Environment:
>Solaris 2.8 with 1G of ram and 2G of swap
>Java 1.3.1
>Lucene 1.2-rc4
>
>Any ideas for eliminating the "out of memory" errors ?
>
>
>
>
>--
>To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
>For additional commands, e-mail:
<ma...@jakarta.apache.org>
--
Winton Davies
Lead Engineer, Overture (NSDQ: OVER)
1820 Gateway Drive, Suite 360
San Mateo, CA 94404
work: (650) 403-2259
cell: (650) 867-1598
http://www.overture.com/
--
To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
For additional commands, e-mail:
<ma...@jakarta.apache.org>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Build index using RAMDirectory "out of memory" errors
Posted by Winton Davies <wd...@cs.stanford.edu>.
java -Xmx1000m ....
Sorry if you already tried resizing your heap. Actually with 1.3.1
you could go up above a gig, but really swapping aint gonna help much.
Winton
>I have been using Lucene for 3 weeks and it rules.
>
>The indexing process can be slow. So I searched the mailgroup archives
>and found example code using RAMDirectory to improve indexing speed.
>The example code I found was indexing 100,000 files at a time to the
>RAMDirectory before writing to disk.
>
>I tried indexing 10,000 files at a time to the RAMDirectory before writing
>to disk. This drastically improved indexing times but sometimes I get
>"out of memory" errors. I am indexing text files and adding 9 fields from
>an Oracle database.
>
>Environment:
>Solaris 2.8 with 1G of ram and 2G of swap
>Java 1.3.1
>Lucene 1.2-rc4
>
>Any ideas for eliminating the "out of memory" errors ?
>
>
>
>
>--
>To unsubscribe, e-mail: <ma...@jakarta.apache.org>
>For additional commands, e-mail: <ma...@jakarta.apache.org>
--
Winton Davies
Lead Engineer, Overture (NSDQ: OVER)
1820 Gateway Drive, Suite 360
San Mateo, CA 94404
work: (650) 403-2259
cell: (650) 867-1598
http://www.overture.com/
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Build index using RAMDirectory "out of memory" errors
Posted by Ian Lea <ia...@blackwell.co.uk>.
Have you tried different values for IndexWriter.mergeFactor?
Setting it to 1000 gave me a 10* speed improvement on some
large index some time ago. Not RAMDirectory though.
Your mileage may vary.
--
Ian.
Kurt Vaag wrote:
>
> I have been using Lucene for 3 weeks and it rules.
>
> The indexing process can be slow. So I searched the mailgroup archives
> and found example code using RAMDirectory to improve indexing speed.
> The example code I found was indexing 100,000 files at a time to the
> RAMDirectory before writing to disk.
>
> I tried indexing 10,000 files at a time to the RAMDirectory before writing
> to disk. This drastically improved indexing times but sometimes I get
> "out of memory" errors. I am indexing text files and adding 9 fields from
> an Oracle database.
>
> Environment:
> Solaris 2.8 with 1G of ram and 2G of swap
> Java 1.3.1
> Lucene 1.2-rc4
>
> Any ideas for eliminating the "out of memory" errors ?
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>