You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Pa...@emainc.com on 2009/09/11 13:57:23 UTC
RE: Indexing large files? - No answers yet...
This issue is still open. Any suggestions/help with this would be
greatly appreciated.
Thanks,
Paul
-----Original Message-----
From: java-user-return-42080-Paul_Murdoch=emainc.com@lucene.apache.org
[mailto:java-user-return-42080-Paul_Murdoch=emainc.com@lucene.apache.org
] On Behalf Of Paul_Murdoch@emainc.com
Sent: Monday, August 31, 2009 10:28 AM
To: java-user@lucene.apache.org
Subject: Indexing large files?
Hi,
I'm working with Lucene 2.4.0 and the JVM (JDK 1.6.0_07). I'm
consistently receiving "OutOfMemoryError: Java heap space", when trying
to index large text files.
Example 1: Indexing a 5 MB text file runs out of memory with a 16 MB
max. heap size. So I increased the max. heap size to 512 MB. This
worked for the 5 MB text file, but Lucene still used 84 MB of heap space
to do this. Why so much?
The class FreqProxTermsWriterPerField appears to be the biggest memory
consumer by far according to JConsole and the TPTP Memory Profiling
plugin for Eclipse Ganymede.
Example 2: Indexing a 62 MB text file runs out of memory with a 512 MB
max. heap size. Increasing the max. heap size to 1024 MB works but
Lucene uses 826 MB of heap space while performing this. Still seems
like way too much memory is being used to do this. I'm sure larger
files would cause the error as it seems correlative.
I'm on a Windows XP SP2 platform with 2 GB of RAM. So what is the best
practice for indexing large files? Here is a code snippet that I'm
using:
// Index the content of a text file.
private Boolean saveTXTFile(File textFile, Document textDocument)
throws CIDBException {
try {
Boolean isFile = textFile.isFile();
Boolean hasTextExtension =
textFile.getName().endsWith(".txt");
if (isFile && hasTextExtension) {
System.out.println("File " +
textFile.getCanonicalPath() + " is being indexed");
Reader textFileReader = new
FileReader(textFile);
if (textDocument == null)
textDocument = new Document();
textDocument.add(new Field("content",
textFileReader));
indexWriter.addDocument(textDocument);
// BREAKS HERE!!!!
}
} catch (FileNotFoundException fnfe) {
System.out.println(fnfe.getMessage());
return false;
} catch (CorruptIndexException cie) {
throw new CIDBException("The index has become
corrupt.");
} catch (IOException ioe) {
System.out.println(ioe.getMessage());
return false;
}
return true;
}
Thanks much,
Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Stopping a runaway search, any ideas?
Posted by Daniel Shane <sh...@LEXUM.UMontreal.CA>.
Wow thats exactly what I was looking for! In the mean time I'll use the
time based collector.
Thanks Uwe and Mark for your help!
Daniel Shane
mark harwood wrote:
> Or https://issues.apache.org/jira/browse/LUCENE-1720 offers lightweight timeout testing at all index access stages prior to calls to Collector e.g. will catch a runaway fuzzy query during it's expensive term expansion phase.
>
>
>
> ----- Original Message ----
> From: Uwe Schindler <uw...@thetaphi.de>
> To: java-user@lucene.apache.org
> Sent: Friday, 11 September, 2009 15:33:19
> Subject: RE: Stopping a runaway search, any ideas?
>
> Yes: TimeLimitedCollector in 2.4.1 (and the new non-deprecated ones in 2.9).
> Just wrap your own collector (like TopDocsCollector) with this class.
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
>> -----Original Message-----
>> From: Daniel Shane [mailto:shaned@LEXUM.UMontreal.CA]
>> Sent: Friday, September 11, 2009 4:26 PM
>> To: java-user@lucene.apache.org
>> Subject: Stopping a runaway search, any ideas?
>>
>> I don't think its possible, but is there something in lucene to cap a
>> search to a predefined time length or is there a way to stop a search
>> when its running for too long?
>>
>> Daniel Shane
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Stopping a runaway search, any ideas?
Posted by mark harwood <ma...@yahoo.co.uk>.
Or https://issues.apache.org/jira/browse/LUCENE-1720 offers lightweight timeout testing at all index access stages prior to calls to Collector e.g. will catch a runaway fuzzy query during it's expensive term expansion phase.
----- Original Message ----
From: Uwe Schindler <uw...@thetaphi.de>
To: java-user@lucene.apache.org
Sent: Friday, 11 September, 2009 15:33:19
Subject: RE: Stopping a runaway search, any ideas?
Yes: TimeLimitedCollector in 2.4.1 (and the new non-deprecated ones in 2.9).
Just wrap your own collector (like TopDocsCollector) with this class.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Daniel Shane [mailto:shaned@LEXUM.UMontreal.CA]
> Sent: Friday, September 11, 2009 4:26 PM
> To: java-user@lucene.apache.org
> Subject: Stopping a runaway search, any ideas?
>
> I don't think its possible, but is there something in lucene to cap a
> search to a predefined time length or is there a way to stop a search
> when its running for too long?
>
> Daniel Shane
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Stopping a runaway search, any ideas?
Posted by Uwe Schindler <uw...@thetaphi.de>.
Yes: TimeLimitedCollector in 2.4.1 (and the new non-deprecated ones in 2.9).
Just wrap your own collector (like TopDocsCollector) with this class.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Daniel Shane [mailto:shaned@LEXUM.UMontreal.CA]
> Sent: Friday, September 11, 2009 4:26 PM
> To: java-user@lucene.apache.org
> Subject: Stopping a runaway search, any ideas?
>
> I don't think its possible, but is there something in lucene to cap a
> search to a predefined time length or is there a way to stop a search
> when its running for too long?
>
> Daniel Shane
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Stopping a runaway search, any ideas?
Posted by Chris Hostetter <ho...@fucit.org>.
: Subject: Stopping a runaway search, any ideas?
: References: <5B...@sc1exc2.corp.emainc.com>
: <5B...@sc1exc2.corp.emainc.com>
: <24098ED350C76D46A4FDBD81B51BE0E903F040B80F@exchange.windows.mmu.acquireme
: dia.com> <5e...@mail.gmail.com>
: In-Reply-To: <5e...@mail.gmail.com>
http://people.apache.org/~hossman/#threadhijack
Thread Hijacking on Mailing Lists
When starting a new discussion on a mailing list, please do not reply to
an existing message, instead start a fresh email. Even if you change the
subject line of your email, other mail headers still track which thread
you replied to and your question is "hidden" in that thread and gets less
attention. It makes following discussions in the mailing list archives
particularly difficult.
See Also: http://en.wikipedia.org/wiki/Thread_hijacking
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Stopping a runaway search, any ideas?
Posted by Daniel Shane <sh...@LEXUM.UMontreal.CA>.
I don't think its possible, but is there something in lucene to cap a
search to a predefined time length or is there a way to stop a search
when its running for too long?
Daniel Shane
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Indexing large files? - No answers yet...
Posted by Brian Pinkerton <br...@lucidimagination.com>.
Quite possibly, but shouldn't one expect Lucene's resource to track
the size of the problem in question? Paul's two examples below use
input files of 5 and 62MB, hardly the size of input I'd expect to
handle in a memory-compromised environment.
bri
On Sep 11, 2009, at 7:43 AM, Glen Newton wrote:
> Paul,
>
> I saw your last post and now understand the issues you face.
>
> I don't think there has been any effort to produce a
> reduced-memory-footprint configurable (RMFC) Lucene. With the many
> mobile devices, embedded and other reduced memory devices, should this
> perhaps be one of the areas the Lucene community looks in to?
>
> -Glen
>
> 2009/9/11 <Pa...@emainc.com>:
>> Thanks Glen!
>>
>> I will take at your project. Unfortunately I will only have 512 MB
>> to 1024 MB to work with as Lucene is only one component in a larger
>> software system running on one machine. I agree with you on the C\C
>> ++ comment. That is what I would normally use for memory intense
>> software. It turns out that the larger file you want to index is
>> the larger the heap space you will need. What I would like to see
>> is a way to "throttle" the indexing process to control the memory
>> footprint. I understand that this will take longer, but if I
>> perform the task during off hours it shouldn't matter. At least the
>> file will be indexed correctly.
>>
>> Thanks,
>> Paul
>>
>>
>> -----Original Message-----
>> From: java-user-return-42272-Paul_Murdoch=emainc.com@lucene.apache.org
>> [mailto:java-user-return-42272-
>> Paul_Murdoch=emainc.com@lucene.apache.org] On Behalf Of Glen Newton
>> Sent: Friday, September 11, 2009 9:53 AM
>> To: java-user@lucene.apache.org
>> Subject: Re: Indexing large files? - No answers yet...
>>
>> In this project:
>> http://zzzoot.blogspot.com/2009/07/project-torngat-building-large-scale.html
>>
>> I concatenate all the text of all of articles of a single journal
>> into
>> a single text file.
>> This can create a text file that is 500MB in size.
>> Lucene is OK in indexing files this size (in parallel even), but I
>> have a heap size of 8GB.
>>
>> I would suggest increasing your heap to as large as your machine can
>> reasonably take.
>> The reality is that Java programs (like Lucene) take up more memory
>> than a similar C or even C++ program.
>> Java may approach C/C++ in speed, but not memory.
>>
>> We don't use Java because of its memory footprint! ;-)
>>
>> See:
>> Programming language shootout: speed:
>> http://shootout.alioth.debian.org/u32q/benchmark.php?test=all&lang=all&d=ndata&calc=calculate&xfullcpu=1&xmem=0&xloc=0&binarytrees=1&chameneosredux=1&fannkuch=1&fasta=1&knucleotide=1&mandelbrot=1&meteor=0&nbody=1&pidigits=1®exdna=1&revcomp=1&spectralnorm=1&threadring=0
>> Programming language shootout: memory:
>> http://shootout.alioth.debian.org/u32q/benchmark.php?test=all&lang=all&d=ndata&calc=calculate&xfullcpu=0&xmem=1&xloc=0&binarytrees=1&chameneosredux=1&fannkuch=1&fasta=1&knucleotide=1&mandelbrot=1&meteor=0&nbody=1&pidigits=1®exdna=1&revcomp=1&spectralnorm=1&threadring=0
>>
>> -glen
>>
>> 2009/9/11 Dan OConnor <do...@acquiremedia.com>:
>>> Paul:
>>>
>>> My first suggestion would be to update your JVM to the latest
>>> version (or at least .14). There were several garbage collection
>>> related issues resolved in version 10 - 13 (especially dealing
>>> with large heaps).
>>>
>>> Next, your IndexWriter parameters would help figure out why you
>>> are using so much RAM
>>> getMaxFieldLength()
>>> getMaxBufferedDocs()
>>> getMaxMergeDocs()
>>> getRAMBufferSizeMB()
>>>
>>> How often are you calling commit?
>>> Do you close your IndexWriter after every document?
>>> How many documents of this size are you indexing?
>>> Have you used luke to look at your index?
>>> If this is a large index, have you optimized it recently?
>>> Are there any searches going on while you are indexing?
>>>
>>>
>>> Regards,
>>> Dan
>>>
>>>
>>> -----Original Message-----
>>> From: Paul_Murdoch@emainc.com [mailto:Paul_Murdoch@emainc.com]
>>> Sent: Friday, September 11, 2009 7:57 AM
>>> To: java-user@lucene.apache.org
>>> Subject: RE: Indexing large files? - No answers yet...
>>>
>>> This issue is still open. Any suggestions/help with this would be
>>> greatly appreciated.
>>>
>>> Thanks,
>>>
>>> Paul
>>>
>>>
>>> -----Original Message-----
>>> From: java-user-return-42080-Paul_Murdoch=emainc.com@lucene.apache.org
>>> [mailto:java-user-return-42080-
>>> Paul_Murdoch=emainc.com@lucene.apache.org
>>> ] On Behalf Of Paul_Murdoch@emainc.com
>>> Sent: Monday, August 31, 2009 10:28 AM
>>> To: java-user@lucene.apache.org
>>> Subject: Indexing large files?
>>>
>>> Hi,
>>>
>>>
>>>
>>> I'm working with Lucene 2.4.0 and the JVM (JDK 1.6.0_07). I'm
>>> consistently receiving "OutOfMemoryError: Java heap space", when
>>> trying
>>> to index large text files.
>>>
>>>
>>>
>>> Example 1: Indexing a 5 MB text file runs out of memory with a 16 MB
>>> max. heap size. So I increased the max. heap size to 512 MB. This
>>> worked for the 5 MB text file, but Lucene still used 84 MB of heap
>>> space
>>> to do this. Why so much?
>>>
>>>
>>>
>>> The class FreqProxTermsWriterPerField appears to be the biggest
>>> memory
>>> consumer by far according to JConsole and the TPTP Memory Profiling
>>> plugin for Eclipse Ganymede.
>>>
>>>
>>>
>>> Example 2: Indexing a 62 MB text file runs out of memory with a
>>> 512 MB
>>> max. heap size. Increasing the max. heap size to 1024 MB works but
>>> Lucene uses 826 MB of heap space while performing this. Still seems
>>> like way too much memory is being used to do this. I'm sure larger
>>> files would cause the error as it seems correlative.
>>>
>>>
>>>
>>> I'm on a Windows XP SP2 platform with 2 GB of RAM. So what is the
>>> best
>>> practice for indexing large files? Here is a code snippet that I'm
>>> using:
>>>
>>>
>>>
>>> // Index the content of a text file.
>>>
>>> private Boolean saveTXTFile(File textFile, Document
>>> textDocument)
>>> throws CIDBException {
>>>
>>>
>>>
>>> try {
>>>
>>>
>>>
>>> Boolean isFile = textFile.isFile();
>>>
>>> Boolean hasTextExtension =
>>> textFile.getName().endsWith(".txt");
>>>
>>>
>>>
>>> if (isFile && hasTextExtension) {
>>>
>>>
>>>
>>> System.out.println("File " +
>>> textFile.getCanonicalPath() + " is being indexed");
>>>
>>> Reader textFileReader = new
>>> FileReader(textFile);
>>>
>>> if (textDocument == null)
>>>
>>> textDocument = new Document();
>>>
>>> textDocument.add(new Field("content",
>>> textFileReader));
>>>
>>> indexWriter.addDocument(textDocument);
>>> // BREAKS HERE!!!!
>>>
>>> }
>>>
>>> } catch (FileNotFoundException fnfe) {
>>>
>>> System.out.println(fnfe.getMessage());
>>>
>>> return false;
>>>
>>> } catch (CorruptIndexException cie) {
>>>
>>> throw new CIDBException("The index has become
>>> corrupt.");
>>>
>>> } catch (IOException ioe) {
>>>
>>> System.out.println(ioe.getMessage());
>>>
>>> return false;
>>>
>>> }
>>>
>>> return true;
>>>
>>> }
>>>
>>>
>>>
>>>
>>>
>>> Thanks much,
>>>
>>>
>>>
>>> Paul
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>>
>>
>> --
>>
>> -
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
>
> --
>
> -
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Indexing large files? - No answers yet...
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Fri, Sep 11, 2009 at 1:15 PM, <Pa...@emainc.com> wrote:
> I've been testing out "paging" the document this past week. I'm
> still working on getting a successful test and think I'm close. The
> down side was a drastic slow down in indexing speed, and lots of
> open files, but that was expected.
You mean a slowdown in indexing speed because you now flush after
every page not after every document, right? That's expected.
But I'm not sure why you'd see a change in the number of open files...
> I tried with small mergeFactors, maxBufferedDocs(haven't tried 1
> though), and ramBufferSizeMB. Using JConsole to monitor the heap
> usage, this method slowly creeps towards my max heap space until
> OOM. I can say that at least some of the document gets indexed
> before OOM. So I performed a heap dump at OOM and saw that
> FreqProxTermsWriterPerField had by far consumed the most memory. I
> haven't looked into that yet...
It's at least ~60 bytes per unique term, not counting the char[] to
hold the term, and FreqProxTermsWriterPerField is exactly where most
of those bytes are allocated (eg its PostingList class).
> Let's say I page the document into ten different smaller documents
> and they are indexed successfully (I'm not quite at this point yet).
> Is there a way to select documents by id and merge them into one
> large document after they are in the index? That was my plan to
> work around OOM and achieve the same end result as trying to index
> the large document in one shot.
You mean at search time right? You basically want the equivalent of
SQL's "group by".
You could make a custom Collector that does this... Or look at how
Solr is iterating on field collapsing
(https://issues.apache.org/jira/browse/SOLR-236)?
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Indexing large files? - No answers yet...
Posted by Pa...@emainc.com.
Thanks Mike!
I've been testing out "paging" the document this past week. I'm still working on getting a successful test and think I'm close. The down side was a drastic slow down in indexing speed, and lots of open files, but that was expected. I tried with small mergeFactors, maxBufferedDocs(haven't tried 1 though), and ramBufferSizeMB. Using JConsole to monitor the heap usage, this method slowly creeps towards my max heap space until OOM. I can say that at least some of the document gets indexed before OOM.
So I performed a heap dump at OOM and saw that FreqProxTermsWriterPerField had by far consumed the most memory. I haven't looked into that yet...
Let's say I page the document into ten different smaller documents and they are indexed successfully (I'm not quite at this point yet). Is there a way to select documents by id and merge them into one large document after they are in the index? That was my plan to work around OOM and achieve the same end result as trying to index the large document in one shot.
Paul
-----Original Message-----
From: java-user-return-42283-Paul_Murdoch=emainc.com@lucene.apache.org [mailto:java-user-return-42283-Paul_Murdoch=emainc.com@lucene.apache.org] On Behalf Of Michael McCandless
Sent: Friday, September 11, 2009 11:54 AM
To: java-user@lucene.apache.org
Subject: Re: Indexing large files? - No answers yet...
To minimize Lucene's RAM usage during indexing, you should flush after
every document, eg by setting the ramBufferSizeMB to something tiny
(or maxBufferedDocs to 1).
But, unfortunately, Lucene cannot flush partway through indexing one
document. Ie, the full document must be indexed into RAM before being
flushed. So the worst case for Lucene will always be a single large
document.
Worse, documents with an unusually high number of unique terms will
then consume even more memory, because there is a certain RAM cost for
each unique term that's seen.
So the absolute worst case is a single large document, all of whose
terms are unique, which seems to be what's being tested here.
In theory one could make a custom indexing chain that knows it will
only hold a single document in ram at once, and could therefore trim
some of the data that we now must store per term, or maybe reduce the
size of the data types (we now use int for most fields per term, but
you could reduce them to shorts and force a flush whenever the shorts
might overflow), etc.
One possible workaround would be to pre-divide such large documents,
before indexing them, though this'd require coalescing at search time.
Mike
On Fri, Sep 11, 2009 at 11:02 AM, <Pa...@emainc.com> wrote:
> Glen,
>
> Absolutely. I think a RMFC Lucene would great, especially for reduced memory or low bandwidth client/server scenarios.
>
> I just looked at your LuSql tool and it just what I needed about 9 months ago :-). I wrote a simple re-indexer that interfaces to an SQL Server 2005 database and Lucene, but I could have saved some time if I knew about LuSql. Unfortunately we're too far down the road in development to test and possibly integrate it into our system now, but I will put it on the R&D list for the next iteration.
>
> Thanks again,
>
> Paul
>
>
> -----Original Message-----
> From: java-user-return-42277-Paul_Murdoch=emainc.com@lucene.apache.org [mailto:java-user-return-42277-Paul_Murdoch=emainc.com@lucene.apache.org] On Behalf Of Glen Newton
> Sent: Friday, September 11, 2009 10:44 AM
> To: java-user@lucene.apache.org
> Subject: Re: Indexing large files? - No answers yet...
>
> Paul,
>
> I saw your last post and now understand the issues you face.
>
> I don't think there has been any effort to produce a
> reduced-memory-footprint configurable (RMFC) Lucene. With the many
> mobile devices, embedded and other reduced memory devices, should this
> perhaps be one of the areas the Lucene community looks in to?
>
> -Glen
>
> 2009/9/11 <Pa...@emainc.com>:
>> Thanks Glen!
>>
>> I will take at your project. Unfortunately I will only have 512 MB to 1024 MB to work with as Lucene is only one component in a larger software system running on one machine. I agree with you on the C\C++ comment. That is what I would normally use for memory intense software. It turns out that the larger file you want to index is the larger the heap space you will need. What I would like to see is a way to "throttle" the indexing process to control the memory footprint. I understand that this will take longer, but if I perform the task during off hours it shouldn't matter. At least the file will be indexed correctly.
>>
>> Thanks,
>> Paul
>>
>>
>> -----Original Message-----
>> From: java-user-return-42272-Paul_Murdoch=emainc.com@lucene.apache.org [mailto:java-user-return-42272-Paul_Murdoch=emainc.com@lucene.apache.org] On Behalf Of Glen Newton
>> Sent: Friday, September 11, 2009 9:53 AM
>> To: java-user@lucene.apache.org
>> Subject: Re: Indexing large files? - No answers yet...
>>
>> In this project:
>> http://zzzoot.blogspot.com/2009/07/project-torngat-building-large-scale.html
>>
>> I concatenate all the text of all of articles of a single journal into
>> a single text file.
>> This can create a text file that is 500MB in size.
>> Lucene is OK in indexing files this size (in parallel even), but I
>> have a heap size of 8GB.
>>
>> I would suggest increasing your heap to as large as your machine can
>> reasonably take.
>> The reality is that Java programs (like Lucene) take up more memory
>> than a similar C or even C++ program.
>> Java may approach C/C++ in speed, but not memory.
>>
>> We don't use Java because of its memory footprint! ;-)
>>
>> See:
>> Programming language shootout: speed:
>> http://shootout.alioth.debian.org/u32q/benchmark.php?test=all&lang=all&d=ndata&calc=calculate&xfullcpu=1&xmem=0&xloc=0&binarytrees=1&chameneosredux=1&fannkuch=1&fasta=1&knucleotide=1&mandelbrot=1&meteor=0&nbody=1&pidigits=1®exdna=1&revcomp=1&spectralnorm=1&threadring=0
>> Programming language shootout: memory:
>> http://shootout.alioth.debian.org/u32q/benchmark.php?test=all&lang=all&d=ndata&calc=calculate&xfullcpu=0&xmem=1&xloc=0&binarytrees=1&chameneosredux=1&fannkuch=1&fasta=1&knucleotide=1&mandelbrot=1&meteor=0&nbody=1&pidigits=1®exdna=1&revcomp=1&spectralnorm=1&threadring=0
>>
>> -glen
>>
>> 2009/9/11 Dan OConnor <do...@acquiremedia.com>:
>>> Paul:
>>>
>>> My first suggestion would be to update your JVM to the latest version (or at least .14). There were several garbage collection related issues resolved in version 10 - 13 (especially dealing with large heaps).
>>>
>>> Next, your IndexWriter parameters would help figure out why you are using so much RAM
>>> getMaxFieldLength()
>>> getMaxBufferedDocs()
>>> getMaxMergeDocs()
>>> getRAMBufferSizeMB()
>>>
>>> How often are you calling commit?
>>> Do you close your IndexWriter after every document?
>>> How many documents of this size are you indexing?
>>> Have you used luke to look at your index?
>>> If this is a large index, have you optimized it recently?
>>> Are there any searches going on while you are indexing?
>>>
>>>
>>> Regards,
>>> Dan
>>>
>>>
>>> -----Original Message-----
>>> From: Paul_Murdoch@emainc.com [mailto:Paul_Murdoch@emainc.com]
>>> Sent: Friday, September 11, 2009 7:57 AM
>>> To: java-user@lucene.apache.org
>>> Subject: RE: Indexing large files? - No answers yet...
>>>
>>> This issue is still open. Any suggestions/help with this would be
>>> greatly appreciated.
>>>
>>> Thanks,
>>>
>>> Paul
>>>
>>>
>>> -----Original Message-----
>>> From: java-user-return-42080-Paul_Murdoch=emainc.com@lucene.apache.org
>>> [mailto:java-user-return-42080-Paul_Murdoch=emainc.com@lucene.apache.org
>>> ] On Behalf Of Paul_Murdoch@emainc.com
>>> Sent: Monday, August 31, 2009 10:28 AM
>>> To: java-user@lucene.apache.org
>>> Subject: Indexing large files?
>>>
>>> Hi,
>>>
>>>
>>>
>>> I'm working with Lucene 2.4.0 and the JVM (JDK 1.6.0_07). I'm
>>> consistently receiving "OutOfMemoryError: Java heap space", when trying
>>> to index large text files.
>>>
>>>
>>>
>>> Example 1: Indexing a 5 MB text file runs out of memory with a 16 MB
>>> max. heap size. So I increased the max. heap size to 512 MB. This
>>> worked for the 5 MB text file, but Lucene still used 84 MB of heap space
>>> to do this. Why so much?
>>>
>>>
>>>
>>> The class FreqProxTermsWriterPerField appears to be the biggest memory
>>> consumer by far according to JConsole and the TPTP Memory Profiling
>>> plugin for Eclipse Ganymede.
>>>
>>>
>>>
>>> Example 2: Indexing a 62 MB text file runs out of memory with a 512 MB
>>> max. heap size. Increasing the max. heap size to 1024 MB works but
>>> Lucene uses 826 MB of heap space while performing this. Still seems
>>> like way too much memory is being used to do this. I'm sure larger
>>> files would cause the error as it seems correlative.
>>>
>>>
>>>
>>> I'm on a Windows XP SP2 platform with 2 GB of RAM. So what is the best
>>> practice for indexing large files? Here is a code snippet that I'm
>>> using:
>>>
>>>
>>>
>>> // Index the content of a text file.
>>>
>>> private Boolean saveTXTFile(File textFile, Document textDocument)
>>> throws CIDBException {
>>>
>>>
>>>
>>> try {
>>>
>>>
>>>
>>> Boolean isFile = textFile.isFile();
>>>
>>> Boolean hasTextExtension =
>>> textFile.getName().endsWith(".txt");
>>>
>>>
>>>
>>> if (isFile && hasTextExtension) {
>>>
>>>
>>>
>>> System.out.println("File " +
>>> textFile.getCanonicalPath() + " is being indexed");
>>>
>>> Reader textFileReader = new
>>> FileReader(textFile);
>>>
>>> if (textDocument == null)
>>>
>>> textDocument = new Document();
>>>
>>> textDocument.add(new Field("content",
>>> textFileReader));
>>>
>>> indexWriter.addDocument(textDocument);
>>> // BREAKS HERE!!!!
>>>
>>> }
>>>
>>> } catch (FileNotFoundException fnfe) {
>>>
>>> System.out.println(fnfe.getMessage());
>>>
>>> return false;
>>>
>>> } catch (CorruptIndexException cie) {
>>>
>>> throw new CIDBException("The index has become
>>> corrupt.");
>>>
>>> } catch (IOException ioe) {
>>>
>>> System.out.println(ioe.getMessage());
>>>
>>> return false;
>>>
>>> }
>>>
>>> return true;
>>>
>>> }
>>>
>>>
>>>
>>>
>>>
>>> Thanks much,
>>>
>>>
>>>
>>> Paul
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>>
>>
>> --
>>
>> -
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
>
> --
>
> -
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Indexing large files? - No answers yet...
Posted by Michael McCandless <lu...@mikemccandless.com>.
To minimize Lucene's RAM usage during indexing, you should flush after
every document, eg by setting the ramBufferSizeMB to something tiny
(or maxBufferedDocs to 1).
But, unfortunately, Lucene cannot flush partway through indexing one
document. Ie, the full document must be indexed into RAM before being
flushed. So the worst case for Lucene will always be a single large
document.
Worse, documents with an unusually high number of unique terms will
then consume even more memory, because there is a certain RAM cost for
each unique term that's seen.
So the absolute worst case is a single large document, all of whose
terms are unique, which seems to be what's being tested here.
In theory one could make a custom indexing chain that knows it will
only hold a single document in ram at once, and could therefore trim
some of the data that we now must store per term, or maybe reduce the
size of the data types (we now use int for most fields per term, but
you could reduce them to shorts and force a flush whenever the shorts
might overflow), etc.
One possible workaround would be to pre-divide such large documents,
before indexing them, though this'd require coalescing at search time.
Mike
On Fri, Sep 11, 2009 at 11:02 AM, <Pa...@emainc.com> wrote:
> Glen,
>
> Absolutely. I think a RMFC Lucene would great, especially for reduced memory or low bandwidth client/server scenarios.
>
> I just looked at your LuSql tool and it just what I needed about 9 months ago :-). I wrote a simple re-indexer that interfaces to an SQL Server 2005 database and Lucene, but I could have saved some time if I knew about LuSql. Unfortunately we're too far down the road in development to test and possibly integrate it into our system now, but I will put it on the R&D list for the next iteration.
>
> Thanks again,
>
> Paul
>
>
> -----Original Message-----
> From: java-user-return-42277-Paul_Murdoch=emainc.com@lucene.apache.org [mailto:java-user-return-42277-Paul_Murdoch=emainc.com@lucene.apache.org] On Behalf Of Glen Newton
> Sent: Friday, September 11, 2009 10:44 AM
> To: java-user@lucene.apache.org
> Subject: Re: Indexing large files? - No answers yet...
>
> Paul,
>
> I saw your last post and now understand the issues you face.
>
> I don't think there has been any effort to produce a
> reduced-memory-footprint configurable (RMFC) Lucene. With the many
> mobile devices, embedded and other reduced memory devices, should this
> perhaps be one of the areas the Lucene community looks in to?
>
> -Glen
>
> 2009/9/11 <Pa...@emainc.com>:
>> Thanks Glen!
>>
>> I will take at your project. Unfortunately I will only have 512 MB to 1024 MB to work with as Lucene is only one component in a larger software system running on one machine. I agree with you on the C\C++ comment. That is what I would normally use for memory intense software. It turns out that the larger file you want to index is the larger the heap space you will need. What I would like to see is a way to "throttle" the indexing process to control the memory footprint. I understand that this will take longer, but if I perform the task during off hours it shouldn't matter. At least the file will be indexed correctly.
>>
>> Thanks,
>> Paul
>>
>>
>> -----Original Message-----
>> From: java-user-return-42272-Paul_Murdoch=emainc.com@lucene.apache.org [mailto:java-user-return-42272-Paul_Murdoch=emainc.com@lucene.apache.org] On Behalf Of Glen Newton
>> Sent: Friday, September 11, 2009 9:53 AM
>> To: java-user@lucene.apache.org
>> Subject: Re: Indexing large files? - No answers yet...
>>
>> In this project:
>> http://zzzoot.blogspot.com/2009/07/project-torngat-building-large-scale.html
>>
>> I concatenate all the text of all of articles of a single journal into
>> a single text file.
>> This can create a text file that is 500MB in size.
>> Lucene is OK in indexing files this size (in parallel even), but I
>> have a heap size of 8GB.
>>
>> I would suggest increasing your heap to as large as your machine can
>> reasonably take.
>> The reality is that Java programs (like Lucene) take up more memory
>> than a similar C or even C++ program.
>> Java may approach C/C++ in speed, but not memory.
>>
>> We don't use Java because of its memory footprint! ;-)
>>
>> See:
>> Programming language shootout: speed:
>> http://shootout.alioth.debian.org/u32q/benchmark.php?test=all&lang=all&d=ndata&calc=calculate&xfullcpu=1&xmem=0&xloc=0&binarytrees=1&chameneosredux=1&fannkuch=1&fasta=1&knucleotide=1&mandelbrot=1&meteor=0&nbody=1&pidigits=1®exdna=1&revcomp=1&spectralnorm=1&threadring=0
>> Programming language shootout: memory:
>> http://shootout.alioth.debian.org/u32q/benchmark.php?test=all&lang=all&d=ndata&calc=calculate&xfullcpu=0&xmem=1&xloc=0&binarytrees=1&chameneosredux=1&fannkuch=1&fasta=1&knucleotide=1&mandelbrot=1&meteor=0&nbody=1&pidigits=1®exdna=1&revcomp=1&spectralnorm=1&threadring=0
>>
>> -glen
>>
>> 2009/9/11 Dan OConnor <do...@acquiremedia.com>:
>>> Paul:
>>>
>>> My first suggestion would be to update your JVM to the latest version (or at least .14). There were several garbage collection related issues resolved in version 10 - 13 (especially dealing with large heaps).
>>>
>>> Next, your IndexWriter parameters would help figure out why you are using so much RAM
>>> getMaxFieldLength()
>>> getMaxBufferedDocs()
>>> getMaxMergeDocs()
>>> getRAMBufferSizeMB()
>>>
>>> How often are you calling commit?
>>> Do you close your IndexWriter after every document?
>>> How many documents of this size are you indexing?
>>> Have you used luke to look at your index?
>>> If this is a large index, have you optimized it recently?
>>> Are there any searches going on while you are indexing?
>>>
>>>
>>> Regards,
>>> Dan
>>>
>>>
>>> -----Original Message-----
>>> From: Paul_Murdoch@emainc.com [mailto:Paul_Murdoch@emainc.com]
>>> Sent: Friday, September 11, 2009 7:57 AM
>>> To: java-user@lucene.apache.org
>>> Subject: RE: Indexing large files? - No answers yet...
>>>
>>> This issue is still open. Any suggestions/help with this would be
>>> greatly appreciated.
>>>
>>> Thanks,
>>>
>>> Paul
>>>
>>>
>>> -----Original Message-----
>>> From: java-user-return-42080-Paul_Murdoch=emainc.com@lucene.apache.org
>>> [mailto:java-user-return-42080-Paul_Murdoch=emainc.com@lucene.apache.org
>>> ] On Behalf Of Paul_Murdoch@emainc.com
>>> Sent: Monday, August 31, 2009 10:28 AM
>>> To: java-user@lucene.apache.org
>>> Subject: Indexing large files?
>>>
>>> Hi,
>>>
>>>
>>>
>>> I'm working with Lucene 2.4.0 and the JVM (JDK 1.6.0_07). I'm
>>> consistently receiving "OutOfMemoryError: Java heap space", when trying
>>> to index large text files.
>>>
>>>
>>>
>>> Example 1: Indexing a 5 MB text file runs out of memory with a 16 MB
>>> max. heap size. So I increased the max. heap size to 512 MB. This
>>> worked for the 5 MB text file, but Lucene still used 84 MB of heap space
>>> to do this. Why so much?
>>>
>>>
>>>
>>> The class FreqProxTermsWriterPerField appears to be the biggest memory
>>> consumer by far according to JConsole and the TPTP Memory Profiling
>>> plugin for Eclipse Ganymede.
>>>
>>>
>>>
>>> Example 2: Indexing a 62 MB text file runs out of memory with a 512 MB
>>> max. heap size. Increasing the max. heap size to 1024 MB works but
>>> Lucene uses 826 MB of heap space while performing this. Still seems
>>> like way too much memory is being used to do this. I'm sure larger
>>> files would cause the error as it seems correlative.
>>>
>>>
>>>
>>> I'm on a Windows XP SP2 platform with 2 GB of RAM. So what is the best
>>> practice for indexing large files? Here is a code snippet that I'm
>>> using:
>>>
>>>
>>>
>>> // Index the content of a text file.
>>>
>>> private Boolean saveTXTFile(File textFile, Document textDocument)
>>> throws CIDBException {
>>>
>>>
>>>
>>> try {
>>>
>>>
>>>
>>> Boolean isFile = textFile.isFile();
>>>
>>> Boolean hasTextExtension =
>>> textFile.getName().endsWith(".txt");
>>>
>>>
>>>
>>> if (isFile && hasTextExtension) {
>>>
>>>
>>>
>>> System.out.println("File " +
>>> textFile.getCanonicalPath() + " is being indexed");
>>>
>>> Reader textFileReader = new
>>> FileReader(textFile);
>>>
>>> if (textDocument == null)
>>>
>>> textDocument = new Document();
>>>
>>> textDocument.add(new Field("content",
>>> textFileReader));
>>>
>>> indexWriter.addDocument(textDocument);
>>> // BREAKS HERE!!!!
>>>
>>> }
>>>
>>> } catch (FileNotFoundException fnfe) {
>>>
>>> System.out.println(fnfe.getMessage());
>>>
>>> return false;
>>>
>>> } catch (CorruptIndexException cie) {
>>>
>>> throw new CIDBException("The index has become
>>> corrupt.");
>>>
>>> } catch (IOException ioe) {
>>>
>>> System.out.println(ioe.getMessage());
>>>
>>> return false;
>>>
>>> }
>>>
>>> return true;
>>>
>>> }
>>>
>>>
>>>
>>>
>>>
>>> Thanks much,
>>>
>>>
>>>
>>> Paul
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>>
>>
>> --
>>
>> -
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
>
> --
>
> -
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Indexing large files? - No answers yet...
Posted by Pa...@emainc.com.
Glen,
Absolutely. I think a RMFC Lucene would great, especially for reduced memory or low bandwidth client/server scenarios.
I just looked at your LuSql tool and it just what I needed about 9 months ago :-). I wrote a simple re-indexer that interfaces to an SQL Server 2005 database and Lucene, but I could have saved some time if I knew about LuSql. Unfortunately we're too far down the road in development to test and possibly integrate it into our system now, but I will put it on the R&D list for the next iteration.
Thanks again,
Paul
-----Original Message-----
From: java-user-return-42277-Paul_Murdoch=emainc.com@lucene.apache.org [mailto:java-user-return-42277-Paul_Murdoch=emainc.com@lucene.apache.org] On Behalf Of Glen Newton
Sent: Friday, September 11, 2009 10:44 AM
To: java-user@lucene.apache.org
Subject: Re: Indexing large files? - No answers yet...
Paul,
I saw your last post and now understand the issues you face.
I don't think there has been any effort to produce a
reduced-memory-footprint configurable (RMFC) Lucene. With the many
mobile devices, embedded and other reduced memory devices, should this
perhaps be one of the areas the Lucene community looks in to?
-Glen
2009/9/11 <Pa...@emainc.com>:
> Thanks Glen!
>
> I will take at your project. Unfortunately I will only have 512 MB to 1024 MB to work with as Lucene is only one component in a larger software system running on one machine. I agree with you on the C\C++ comment. That is what I would normally use for memory intense software. It turns out that the larger file you want to index is the larger the heap space you will need. What I would like to see is a way to "throttle" the indexing process to control the memory footprint. I understand that this will take longer, but if I perform the task during off hours it shouldn't matter. At least the file will be indexed correctly.
>
> Thanks,
> Paul
>
>
> -----Original Message-----
> From: java-user-return-42272-Paul_Murdoch=emainc.com@lucene.apache.org [mailto:java-user-return-42272-Paul_Murdoch=emainc.com@lucene.apache.org] On Behalf Of Glen Newton
> Sent: Friday, September 11, 2009 9:53 AM
> To: java-user@lucene.apache.org
> Subject: Re: Indexing large files? - No answers yet...
>
> In this project:
> http://zzzoot.blogspot.com/2009/07/project-torngat-building-large-scale.html
>
> I concatenate all the text of all of articles of a single journal into
> a single text file.
> This can create a text file that is 500MB in size.
> Lucene is OK in indexing files this size (in parallel even), but I
> have a heap size of 8GB.
>
> I would suggest increasing your heap to as large as your machine can
> reasonably take.
> The reality is that Java programs (like Lucene) take up more memory
> than a similar C or even C++ program.
> Java may approach C/C++ in speed, but not memory.
>
> We don't use Java because of its memory footprint! ;-)
>
> See:
> Programming language shootout: speed:
> http://shootout.alioth.debian.org/u32q/benchmark.php?test=all&lang=all&d=ndata&calc=calculate&xfullcpu=1&xmem=0&xloc=0&binarytrees=1&chameneosredux=1&fannkuch=1&fasta=1&knucleotide=1&mandelbrot=1&meteor=0&nbody=1&pidigits=1®exdna=1&revcomp=1&spectralnorm=1&threadring=0
> Programming language shootout: memory:
> http://shootout.alioth.debian.org/u32q/benchmark.php?test=all&lang=all&d=ndata&calc=calculate&xfullcpu=0&xmem=1&xloc=0&binarytrees=1&chameneosredux=1&fannkuch=1&fasta=1&knucleotide=1&mandelbrot=1&meteor=0&nbody=1&pidigits=1®exdna=1&revcomp=1&spectralnorm=1&threadring=0
>
> -glen
>
> 2009/9/11 Dan OConnor <do...@acquiremedia.com>:
>> Paul:
>>
>> My first suggestion would be to update your JVM to the latest version (or at least .14). There were several garbage collection related issues resolved in version 10 - 13 (especially dealing with large heaps).
>>
>> Next, your IndexWriter parameters would help figure out why you are using so much RAM
>> getMaxFieldLength()
>> getMaxBufferedDocs()
>> getMaxMergeDocs()
>> getRAMBufferSizeMB()
>>
>> How often are you calling commit?
>> Do you close your IndexWriter after every document?
>> How many documents of this size are you indexing?
>> Have you used luke to look at your index?
>> If this is a large index, have you optimized it recently?
>> Are there any searches going on while you are indexing?
>>
>>
>> Regards,
>> Dan
>>
>>
>> -----Original Message-----
>> From: Paul_Murdoch@emainc.com [mailto:Paul_Murdoch@emainc.com]
>> Sent: Friday, September 11, 2009 7:57 AM
>> To: java-user@lucene.apache.org
>> Subject: RE: Indexing large files? - No answers yet...
>>
>> This issue is still open. Any suggestions/help with this would be
>> greatly appreciated.
>>
>> Thanks,
>>
>> Paul
>>
>>
>> -----Original Message-----
>> From: java-user-return-42080-Paul_Murdoch=emainc.com@lucene.apache.org
>> [mailto:java-user-return-42080-Paul_Murdoch=emainc.com@lucene.apache.org
>> ] On Behalf Of Paul_Murdoch@emainc.com
>> Sent: Monday, August 31, 2009 10:28 AM
>> To: java-user@lucene.apache.org
>> Subject: Indexing large files?
>>
>> Hi,
>>
>>
>>
>> I'm working with Lucene 2.4.0 and the JVM (JDK 1.6.0_07). I'm
>> consistently receiving "OutOfMemoryError: Java heap space", when trying
>> to index large text files.
>>
>>
>>
>> Example 1: Indexing a 5 MB text file runs out of memory with a 16 MB
>> max. heap size. So I increased the max. heap size to 512 MB. This
>> worked for the 5 MB text file, but Lucene still used 84 MB of heap space
>> to do this. Why so much?
>>
>>
>>
>> The class FreqProxTermsWriterPerField appears to be the biggest memory
>> consumer by far according to JConsole and the TPTP Memory Profiling
>> plugin for Eclipse Ganymede.
>>
>>
>>
>> Example 2: Indexing a 62 MB text file runs out of memory with a 512 MB
>> max. heap size. Increasing the max. heap size to 1024 MB works but
>> Lucene uses 826 MB of heap space while performing this. Still seems
>> like way too much memory is being used to do this. I'm sure larger
>> files would cause the error as it seems correlative.
>>
>>
>>
>> I'm on a Windows XP SP2 platform with 2 GB of RAM. So what is the best
>> practice for indexing large files? Here is a code snippet that I'm
>> using:
>>
>>
>>
>> // Index the content of a text file.
>>
>> private Boolean saveTXTFile(File textFile, Document textDocument)
>> throws CIDBException {
>>
>>
>>
>> try {
>>
>>
>>
>> Boolean isFile = textFile.isFile();
>>
>> Boolean hasTextExtension =
>> textFile.getName().endsWith(".txt");
>>
>>
>>
>> if (isFile && hasTextExtension) {
>>
>>
>>
>> System.out.println("File " +
>> textFile.getCanonicalPath() + " is being indexed");
>>
>> Reader textFileReader = new
>> FileReader(textFile);
>>
>> if (textDocument == null)
>>
>> textDocument = new Document();
>>
>> textDocument.add(new Field("content",
>> textFileReader));
>>
>> indexWriter.addDocument(textDocument);
>> // BREAKS HERE!!!!
>>
>> }
>>
>> } catch (FileNotFoundException fnfe) {
>>
>> System.out.println(fnfe.getMessage());
>>
>> return false;
>>
>> } catch (CorruptIndexException cie) {
>>
>> throw new CIDBException("The index has become
>> corrupt.");
>>
>> } catch (IOException ioe) {
>>
>> System.out.println(ioe.getMessage());
>>
>> return false;
>>
>> }
>>
>> return true;
>>
>> }
>>
>>
>>
>>
>>
>> Thanks much,
>>
>>
>>
>> Paul
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
>
> --
>
> -
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
-
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Indexing large files? - No answers yet...
Posted by Glen Newton <gl...@gmail.com>.
Paul,
I saw your last post and now understand the issues you face.
I don't think there has been any effort to produce a
reduced-memory-footprint configurable (RMFC) Lucene. With the many
mobile devices, embedded and other reduced memory devices, should this
perhaps be one of the areas the Lucene community looks in to?
-Glen
2009/9/11 <Pa...@emainc.com>:
> Thanks Glen!
>
> I will take at your project. Unfortunately I will only have 512 MB to 1024 MB to work with as Lucene is only one component in a larger software system running on one machine. I agree with you on the C\C++ comment. That is what I would normally use for memory intense software. It turns out that the larger file you want to index is the larger the heap space you will need. What I would like to see is a way to "throttle" the indexing process to control the memory footprint. I understand that this will take longer, but if I perform the task during off hours it shouldn't matter. At least the file will be indexed correctly.
>
> Thanks,
> Paul
>
>
> -----Original Message-----
> From: java-user-return-42272-Paul_Murdoch=emainc.com@lucene.apache.org [mailto:java-user-return-42272-Paul_Murdoch=emainc.com@lucene.apache.org] On Behalf Of Glen Newton
> Sent: Friday, September 11, 2009 9:53 AM
> To: java-user@lucene.apache.org
> Subject: Re: Indexing large files? - No answers yet...
>
> In this project:
> http://zzzoot.blogspot.com/2009/07/project-torngat-building-large-scale.html
>
> I concatenate all the text of all of articles of a single journal into
> a single text file.
> This can create a text file that is 500MB in size.
> Lucene is OK in indexing files this size (in parallel even), but I
> have a heap size of 8GB.
>
> I would suggest increasing your heap to as large as your machine can
> reasonably take.
> The reality is that Java programs (like Lucene) take up more memory
> than a similar C or even C++ program.
> Java may approach C/C++ in speed, but not memory.
>
> We don't use Java because of its memory footprint! ;-)
>
> See:
> Programming language shootout: speed:
> http://shootout.alioth.debian.org/u32q/benchmark.php?test=all&lang=all&d=ndata&calc=calculate&xfullcpu=1&xmem=0&xloc=0&binarytrees=1&chameneosredux=1&fannkuch=1&fasta=1&knucleotide=1&mandelbrot=1&meteor=0&nbody=1&pidigits=1®exdna=1&revcomp=1&spectralnorm=1&threadring=0
> Programming language shootout: memory:
> http://shootout.alioth.debian.org/u32q/benchmark.php?test=all&lang=all&d=ndata&calc=calculate&xfullcpu=0&xmem=1&xloc=0&binarytrees=1&chameneosredux=1&fannkuch=1&fasta=1&knucleotide=1&mandelbrot=1&meteor=0&nbody=1&pidigits=1®exdna=1&revcomp=1&spectralnorm=1&threadring=0
>
> -glen
>
> 2009/9/11 Dan OConnor <do...@acquiremedia.com>:
>> Paul:
>>
>> My first suggestion would be to update your JVM to the latest version (or at least .14). There were several garbage collection related issues resolved in version 10 - 13 (especially dealing with large heaps).
>>
>> Next, your IndexWriter parameters would help figure out why you are using so much RAM
>> getMaxFieldLength()
>> getMaxBufferedDocs()
>> getMaxMergeDocs()
>> getRAMBufferSizeMB()
>>
>> How often are you calling commit?
>> Do you close your IndexWriter after every document?
>> How many documents of this size are you indexing?
>> Have you used luke to look at your index?
>> If this is a large index, have you optimized it recently?
>> Are there any searches going on while you are indexing?
>>
>>
>> Regards,
>> Dan
>>
>>
>> -----Original Message-----
>> From: Paul_Murdoch@emainc.com [mailto:Paul_Murdoch@emainc.com]
>> Sent: Friday, September 11, 2009 7:57 AM
>> To: java-user@lucene.apache.org
>> Subject: RE: Indexing large files? - No answers yet...
>>
>> This issue is still open. Any suggestions/help with this would be
>> greatly appreciated.
>>
>> Thanks,
>>
>> Paul
>>
>>
>> -----Original Message-----
>> From: java-user-return-42080-Paul_Murdoch=emainc.com@lucene.apache.org
>> [mailto:java-user-return-42080-Paul_Murdoch=emainc.com@lucene.apache.org
>> ] On Behalf Of Paul_Murdoch@emainc.com
>> Sent: Monday, August 31, 2009 10:28 AM
>> To: java-user@lucene.apache.org
>> Subject: Indexing large files?
>>
>> Hi,
>>
>>
>>
>> I'm working with Lucene 2.4.0 and the JVM (JDK 1.6.0_07). I'm
>> consistently receiving "OutOfMemoryError: Java heap space", when trying
>> to index large text files.
>>
>>
>>
>> Example 1: Indexing a 5 MB text file runs out of memory with a 16 MB
>> max. heap size. So I increased the max. heap size to 512 MB. This
>> worked for the 5 MB text file, but Lucene still used 84 MB of heap space
>> to do this. Why so much?
>>
>>
>>
>> The class FreqProxTermsWriterPerField appears to be the biggest memory
>> consumer by far according to JConsole and the TPTP Memory Profiling
>> plugin for Eclipse Ganymede.
>>
>>
>>
>> Example 2: Indexing a 62 MB text file runs out of memory with a 512 MB
>> max. heap size. Increasing the max. heap size to 1024 MB works but
>> Lucene uses 826 MB of heap space while performing this. Still seems
>> like way too much memory is being used to do this. I'm sure larger
>> files would cause the error as it seems correlative.
>>
>>
>>
>> I'm on a Windows XP SP2 platform with 2 GB of RAM. So what is the best
>> practice for indexing large files? Here is a code snippet that I'm
>> using:
>>
>>
>>
>> // Index the content of a text file.
>>
>> private Boolean saveTXTFile(File textFile, Document textDocument)
>> throws CIDBException {
>>
>>
>>
>> try {
>>
>>
>>
>> Boolean isFile = textFile.isFile();
>>
>> Boolean hasTextExtension =
>> textFile.getName().endsWith(".txt");
>>
>>
>>
>> if (isFile && hasTextExtension) {
>>
>>
>>
>> System.out.println("File " +
>> textFile.getCanonicalPath() + " is being indexed");
>>
>> Reader textFileReader = new
>> FileReader(textFile);
>>
>> if (textDocument == null)
>>
>> textDocument = new Document();
>>
>> textDocument.add(new Field("content",
>> textFileReader));
>>
>> indexWriter.addDocument(textDocument);
>> // BREAKS HERE!!!!
>>
>> }
>>
>> } catch (FileNotFoundException fnfe) {
>>
>> System.out.println(fnfe.getMessage());
>>
>> return false;
>>
>> } catch (CorruptIndexException cie) {
>>
>> throw new CIDBException("The index has become
>> corrupt.");
>>
>> } catch (IOException ioe) {
>>
>> System.out.println(ioe.getMessage());
>>
>> return false;
>>
>> }
>>
>> return true;
>>
>> }
>>
>>
>>
>>
>>
>> Thanks much,
>>
>>
>>
>> Paul
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
>
> --
>
> -
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
-
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Indexing large files? - No answers yet...
Posted by Pa...@emainc.com.
Thanks Glen!
I will take at your project. Unfortunately I will only have 512 MB to 1024 MB to work with as Lucene is only one component in a larger software system running on one machine. I agree with you on the C\C++ comment. That is what I would normally use for memory intense software. It turns out that the larger file you want to index is the larger the heap space you will need. What I would like to see is a way to "throttle" the indexing process to control the memory footprint. I understand that this will take longer, but if I perform the task during off hours it shouldn't matter. At least the file will be indexed correctly.
Thanks,
Paul
-----Original Message-----
From: java-user-return-42272-Paul_Murdoch=emainc.com@lucene.apache.org [mailto:java-user-return-42272-Paul_Murdoch=emainc.com@lucene.apache.org] On Behalf Of Glen Newton
Sent: Friday, September 11, 2009 9:53 AM
To: java-user@lucene.apache.org
Subject: Re: Indexing large files? - No answers yet...
In this project:
http://zzzoot.blogspot.com/2009/07/project-torngat-building-large-scale.html
I concatenate all the text of all of articles of a single journal into
a single text file.
This can create a text file that is 500MB in size.
Lucene is OK in indexing files this size (in parallel even), but I
have a heap size of 8GB.
I would suggest increasing your heap to as large as your machine can
reasonably take.
The reality is that Java programs (like Lucene) take up more memory
than a similar C or even C++ program.
Java may approach C/C++ in speed, but not memory.
We don't use Java because of its memory footprint! ;-)
See:
Programming language shootout: speed:
http://shootout.alioth.debian.org/u32q/benchmark.php?test=all&lang=all&d=ndata&calc=calculate&xfullcpu=1&xmem=0&xloc=0&binarytrees=1&chameneosredux=1&fannkuch=1&fasta=1&knucleotide=1&mandelbrot=1&meteor=0&nbody=1&pidigits=1®exdna=1&revcomp=1&spectralnorm=1&threadring=0
Programming language shootout: memory:
http://shootout.alioth.debian.org/u32q/benchmark.php?test=all&lang=all&d=ndata&calc=calculate&xfullcpu=0&xmem=1&xloc=0&binarytrees=1&chameneosredux=1&fannkuch=1&fasta=1&knucleotide=1&mandelbrot=1&meteor=0&nbody=1&pidigits=1®exdna=1&revcomp=1&spectralnorm=1&threadring=0
-glen
2009/9/11 Dan OConnor <do...@acquiremedia.com>:
> Paul:
>
> My first suggestion would be to update your JVM to the latest version (or at least .14). There were several garbage collection related issues resolved in version 10 - 13 (especially dealing with large heaps).
>
> Next, your IndexWriter parameters would help figure out why you are using so much RAM
> getMaxFieldLength()
> getMaxBufferedDocs()
> getMaxMergeDocs()
> getRAMBufferSizeMB()
>
> How often are you calling commit?
> Do you close your IndexWriter after every document?
> How many documents of this size are you indexing?
> Have you used luke to look at your index?
> If this is a large index, have you optimized it recently?
> Are there any searches going on while you are indexing?
>
>
> Regards,
> Dan
>
>
> -----Original Message-----
> From: Paul_Murdoch@emainc.com [mailto:Paul_Murdoch@emainc.com]
> Sent: Friday, September 11, 2009 7:57 AM
> To: java-user@lucene.apache.org
> Subject: RE: Indexing large files? - No answers yet...
>
> This issue is still open. Any suggestions/help with this would be
> greatly appreciated.
>
> Thanks,
>
> Paul
>
>
> -----Original Message-----
> From: java-user-return-42080-Paul_Murdoch=emainc.com@lucene.apache.org
> [mailto:java-user-return-42080-Paul_Murdoch=emainc.com@lucene.apache.org
> ] On Behalf Of Paul_Murdoch@emainc.com
> Sent: Monday, August 31, 2009 10:28 AM
> To: java-user@lucene.apache.org
> Subject: Indexing large files?
>
> Hi,
>
>
>
> I'm working with Lucene 2.4.0 and the JVM (JDK 1.6.0_07). I'm
> consistently receiving "OutOfMemoryError: Java heap space", when trying
> to index large text files.
>
>
>
> Example 1: Indexing a 5 MB text file runs out of memory with a 16 MB
> max. heap size. So I increased the max. heap size to 512 MB. This
> worked for the 5 MB text file, but Lucene still used 84 MB of heap space
> to do this. Why so much?
>
>
>
> The class FreqProxTermsWriterPerField appears to be the biggest memory
> consumer by far according to JConsole and the TPTP Memory Profiling
> plugin for Eclipse Ganymede.
>
>
>
> Example 2: Indexing a 62 MB text file runs out of memory with a 512 MB
> max. heap size. Increasing the max. heap size to 1024 MB works but
> Lucene uses 826 MB of heap space while performing this. Still seems
> like way too much memory is being used to do this. I'm sure larger
> files would cause the error as it seems correlative.
>
>
>
> I'm on a Windows XP SP2 platform with 2 GB of RAM. So what is the best
> practice for indexing large files? Here is a code snippet that I'm
> using:
>
>
>
> // Index the content of a text file.
>
> private Boolean saveTXTFile(File textFile, Document textDocument)
> throws CIDBException {
>
>
>
> try {
>
>
>
> Boolean isFile = textFile.isFile();
>
> Boolean hasTextExtension =
> textFile.getName().endsWith(".txt");
>
>
>
> if (isFile && hasTextExtension) {
>
>
>
> System.out.println("File " +
> textFile.getCanonicalPath() + " is being indexed");
>
> Reader textFileReader = new
> FileReader(textFile);
>
> if (textDocument == null)
>
> textDocument = new Document();
>
> textDocument.add(new Field("content",
> textFileReader));
>
> indexWriter.addDocument(textDocument);
> // BREAKS HERE!!!!
>
> }
>
> } catch (FileNotFoundException fnfe) {
>
> System.out.println(fnfe.getMessage());
>
> return false;
>
> } catch (CorruptIndexException cie) {
>
> throw new CIDBException("The index has become
> corrupt.");
>
> } catch (IOException ioe) {
>
> System.out.println(ioe.getMessage());
>
> return false;
>
> }
>
> return true;
>
> }
>
>
>
>
>
> Thanks much,
>
>
>
> Paul
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
-
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Indexing large files? - No answers yet...
Posted by Glen Newton <gl...@gmail.com>.
In this project:
http://zzzoot.blogspot.com/2009/07/project-torngat-building-large-scale.html
I concatenate all the text of all of articles of a single journal into
a single text file.
This can create a text file that is 500MB in size.
Lucene is OK in indexing files this size (in parallel even), but I
have a heap size of 8GB.
I would suggest increasing your heap to as large as your machine can
reasonably take.
The reality is that Java programs (like Lucene) take up more memory
than a similar C or even C++ program.
Java may approach C/C++ in speed, but not memory.
We don't use Java because of its memory footprint! ;-)
See:
Programming language shootout: speed:
http://shootout.alioth.debian.org/u32q/benchmark.php?test=all&lang=all&d=ndata&calc=calculate&xfullcpu=1&xmem=0&xloc=0&binarytrees=1&chameneosredux=1&fannkuch=1&fasta=1&knucleotide=1&mandelbrot=1&meteor=0&nbody=1&pidigits=1®exdna=1&revcomp=1&spectralnorm=1&threadring=0
Programming language shootout: memory:
http://shootout.alioth.debian.org/u32q/benchmark.php?test=all&lang=all&d=ndata&calc=calculate&xfullcpu=0&xmem=1&xloc=0&binarytrees=1&chameneosredux=1&fannkuch=1&fasta=1&knucleotide=1&mandelbrot=1&meteor=0&nbody=1&pidigits=1®exdna=1&revcomp=1&spectralnorm=1&threadring=0
-glen
2009/9/11 Dan OConnor <do...@acquiremedia.com>:
> Paul:
>
> My first suggestion would be to update your JVM to the latest version (or at least .14). There were several garbage collection related issues resolved in version 10 - 13 (especially dealing with large heaps).
>
> Next, your IndexWriter parameters would help figure out why you are using so much RAM
> getMaxFieldLength()
> getMaxBufferedDocs()
> getMaxMergeDocs()
> getRAMBufferSizeMB()
>
> How often are you calling commit?
> Do you close your IndexWriter after every document?
> How many documents of this size are you indexing?
> Have you used luke to look at your index?
> If this is a large index, have you optimized it recently?
> Are there any searches going on while you are indexing?
>
>
> Regards,
> Dan
>
>
> -----Original Message-----
> From: Paul_Murdoch@emainc.com [mailto:Paul_Murdoch@emainc.com]
> Sent: Friday, September 11, 2009 7:57 AM
> To: java-user@lucene.apache.org
> Subject: RE: Indexing large files? - No answers yet...
>
> This issue is still open. Any suggestions/help with this would be
> greatly appreciated.
>
> Thanks,
>
> Paul
>
>
> -----Original Message-----
> From: java-user-return-42080-Paul_Murdoch=emainc.com@lucene.apache.org
> [mailto:java-user-return-42080-Paul_Murdoch=emainc.com@lucene.apache.org
> ] On Behalf Of Paul_Murdoch@emainc.com
> Sent: Monday, August 31, 2009 10:28 AM
> To: java-user@lucene.apache.org
> Subject: Indexing large files?
>
> Hi,
>
>
>
> I'm working with Lucene 2.4.0 and the JVM (JDK 1.6.0_07). I'm
> consistently receiving "OutOfMemoryError: Java heap space", when trying
> to index large text files.
>
>
>
> Example 1: Indexing a 5 MB text file runs out of memory with a 16 MB
> max. heap size. So I increased the max. heap size to 512 MB. This
> worked for the 5 MB text file, but Lucene still used 84 MB of heap space
> to do this. Why so much?
>
>
>
> The class FreqProxTermsWriterPerField appears to be the biggest memory
> consumer by far according to JConsole and the TPTP Memory Profiling
> plugin for Eclipse Ganymede.
>
>
>
> Example 2: Indexing a 62 MB text file runs out of memory with a 512 MB
> max. heap size. Increasing the max. heap size to 1024 MB works but
> Lucene uses 826 MB of heap space while performing this. Still seems
> like way too much memory is being used to do this. I'm sure larger
> files would cause the error as it seems correlative.
>
>
>
> I'm on a Windows XP SP2 platform with 2 GB of RAM. So what is the best
> practice for indexing large files? Here is a code snippet that I'm
> using:
>
>
>
> // Index the content of a text file.
>
> private Boolean saveTXTFile(File textFile, Document textDocument)
> throws CIDBException {
>
>
>
> try {
>
>
>
> Boolean isFile = textFile.isFile();
>
> Boolean hasTextExtension =
> textFile.getName().endsWith(".txt");
>
>
>
> if (isFile && hasTextExtension) {
>
>
>
> System.out.println("File " +
> textFile.getCanonicalPath() + " is being indexed");
>
> Reader textFileReader = new
> FileReader(textFile);
>
> if (textDocument == null)
>
> textDocument = new Document();
>
> textDocument.add(new Field("content",
> textFileReader));
>
> indexWriter.addDocument(textDocument);
> // BREAKS HERE!!!!
>
> }
>
> } catch (FileNotFoundException fnfe) {
>
> System.out.println(fnfe.getMessage());
>
> return false;
>
> } catch (CorruptIndexException cie) {
>
> throw new CIDBException("The index has become
> corrupt.");
>
> } catch (IOException ioe) {
>
> System.out.println(ioe.getMessage());
>
> return false;
>
> }
>
> return true;
>
> }
>
>
>
>
>
> Thanks much,
>
>
>
> Paul
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
-
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Indexing large files? - No answers yet...
Posted by Pa...@emainc.com.
Thanks Dan!
I upgraded my JVM from .12 to .16. I'll test with that.
I've been testing by setting many IndexWriter parameters manually to see
where the best performance is. Then net result was just delaying the
OOM.
The scenario is a test with an empty index. I have a 5 MB file with
800,000 unique terms in it. I make one document with the file and then
add that document to the IndexWriter for indexing. At a heap space of
64 MB the OOM occurs almost immediately when the
IndexWriter.add(document) is called. If I increase the heap space to 128
MB indexing is successful taking less than 5 seconds to complete. So...
I commit only once.
The OOM occurs before I can close the IndexWriter at 64 MB heap space. I
close and optimize on the successful 128 MB test.
I'm only indexing one document.
I use Luke all the time for other indexes, but after the OOM on this
test not even the "Force Open" will get me into the index.
There are no searches going on. This is a test to try and index one 5
MB file to an empty index.
So you're probably asking why I don't just increase the heap space and
be happy. The answer is that the larger the file the more heap space is
needed. The system I'm developing doesn't have the heap space required
for the potential large files that the end user might try to index. So
I would like a way to index the file as one document using a small
memory footprint. It would be nice to be able to "throttle" the
indexing of large files to control memory usage.
Thanks,
Paul
-----Original Message-----
From: java-user-return-42271-Paul_Murdoch=emainc.com@lucene.apache.org
[mailto:java-user-return-42271-Paul_Murdoch=emainc.com@lucene.apache.org
] On Behalf Of Dan OConnor
Sent: Friday, September 11, 2009 8:13 AM
To: java-user@lucene.apache.org
Subject: RE: Indexing large files? - No answers yet...
Paul:
My first suggestion would be to update your JVM to the latest version
(or at least .14). There were several garbage collection related issues
resolved in version 10 - 13 (especially dealing with large heaps).
Next, your IndexWriter parameters would help figure out why you are
using so much RAM
getMaxFieldLength()
getMaxBufferedDocs()
getMaxMergeDocs()
getRAMBufferSizeMB()
How often are you calling commit?
Do you close your IndexWriter after every document?
How many documents of this size are you indexing?
Have you used luke to look at your index?
If this is a large index, have you optimized it recently?
Are there any searches going on while you are indexing?
Regards,
Dan
-----Original Message-----
From: Paul_Murdoch@emainc.com [mailto:Paul_Murdoch@emainc.com]
Sent: Friday, September 11, 2009 7:57 AM
To: java-user@lucene.apache.org
Subject: RE: Indexing large files? - No answers yet...
This issue is still open. Any suggestions/help with this would be
greatly appreciated.
Thanks,
Paul
-----Original Message-----
From: java-user-return-42080-Paul_Murdoch=emainc.com@lucene.apache.org
[mailto:java-user-return-42080-Paul_Murdoch=emainc.com@lucene.apache.org
] On Behalf Of Paul_Murdoch@emainc.com
Sent: Monday, August 31, 2009 10:28 AM
To: java-user@lucene.apache.org
Subject: Indexing large files?
Hi,
I'm working with Lucene 2.4.0 and the JVM (JDK 1.6.0_07). I'm
consistently receiving "OutOfMemoryError: Java heap space", when trying
to index large text files.
Example 1: Indexing a 5 MB text file runs out of memory with a 16 MB
max. heap size. So I increased the max. heap size to 512 MB. This
worked for the 5 MB text file, but Lucene still used 84 MB of heap space
to do this. Why so much?
The class FreqProxTermsWriterPerField appears to be the biggest memory
consumer by far according to JConsole and the TPTP Memory Profiling
plugin for Eclipse Ganymede.
Example 2: Indexing a 62 MB text file runs out of memory with a 512 MB
max. heap size. Increasing the max. heap size to 1024 MB works but
Lucene uses 826 MB of heap space while performing this. Still seems
like way too much memory is being used to do this. I'm sure larger
files would cause the error as it seems correlative.
I'm on a Windows XP SP2 platform with 2 GB of RAM. So what is the best
practice for indexing large files? Here is a code snippet that I'm
using:
// Index the content of a text file.
private Boolean saveTXTFile(File textFile, Document textDocument)
throws CIDBException {
try {
Boolean isFile = textFile.isFile();
Boolean hasTextExtension =
textFile.getName().endsWith(".txt");
if (isFile && hasTextExtension) {
System.out.println("File " +
textFile.getCanonicalPath() + " is being indexed");
Reader textFileReader = new
FileReader(textFile);
if (textDocument == null)
textDocument = new Document();
textDocument.add(new Field("content",
textFileReader));
indexWriter.addDocument(textDocument);
// BREAKS HERE!!!!
}
} catch (FileNotFoundException fnfe) {
System.out.println(fnfe.getMessage());
return false;
} catch (CorruptIndexException cie) {
throw new CIDBException("The index has become
corrupt.");
} catch (IOException ioe) {
System.out.println(ioe.getMessage());
return false;
}
return true;
}
Thanks much,
Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Indexing large files? - No answers yet...
Posted by Dan OConnor <do...@acquiremedia.com>.
Paul:
My first suggestion would be to update your JVM to the latest version (or at least .14). There were several garbage collection related issues resolved in version 10 - 13 (especially dealing with large heaps).
Next, your IndexWriter parameters would help figure out why you are using so much RAM
getMaxFieldLength()
getMaxBufferedDocs()
getMaxMergeDocs()
getRAMBufferSizeMB()
How often are you calling commit?
Do you close your IndexWriter after every document?
How many documents of this size are you indexing?
Have you used luke to look at your index?
If this is a large index, have you optimized it recently?
Are there any searches going on while you are indexing?
Regards,
Dan
-----Original Message-----
From: Paul_Murdoch@emainc.com [mailto:Paul_Murdoch@emainc.com]
Sent: Friday, September 11, 2009 7:57 AM
To: java-user@lucene.apache.org
Subject: RE: Indexing large files? - No answers yet...
This issue is still open. Any suggestions/help with this would be
greatly appreciated.
Thanks,
Paul
-----Original Message-----
From: java-user-return-42080-Paul_Murdoch=emainc.com@lucene.apache.org
[mailto:java-user-return-42080-Paul_Murdoch=emainc.com@lucene.apache.org
] On Behalf Of Paul_Murdoch@emainc.com
Sent: Monday, August 31, 2009 10:28 AM
To: java-user@lucene.apache.org
Subject: Indexing large files?
Hi,
I'm working with Lucene 2.4.0 and the JVM (JDK 1.6.0_07). I'm
consistently receiving "OutOfMemoryError: Java heap space", when trying
to index large text files.
Example 1: Indexing a 5 MB text file runs out of memory with a 16 MB
max. heap size. So I increased the max. heap size to 512 MB. This
worked for the 5 MB text file, but Lucene still used 84 MB of heap space
to do this. Why so much?
The class FreqProxTermsWriterPerField appears to be the biggest memory
consumer by far according to JConsole and the TPTP Memory Profiling
plugin for Eclipse Ganymede.
Example 2: Indexing a 62 MB text file runs out of memory with a 512 MB
max. heap size. Increasing the max. heap size to 1024 MB works but
Lucene uses 826 MB of heap space while performing this. Still seems
like way too much memory is being used to do this. I'm sure larger
files would cause the error as it seems correlative.
I'm on a Windows XP SP2 platform with 2 GB of RAM. So what is the best
practice for indexing large files? Here is a code snippet that I'm
using:
// Index the content of a text file.
private Boolean saveTXTFile(File textFile, Document textDocument)
throws CIDBException {
try {
Boolean isFile = textFile.isFile();
Boolean hasTextExtension =
textFile.getName().endsWith(".txt");
if (isFile && hasTextExtension) {
System.out.println("File " +
textFile.getCanonicalPath() + " is being indexed");
Reader textFileReader = new
FileReader(textFile);
if (textDocument == null)
textDocument = new Document();
textDocument.add(new Field("content",
textFileReader));
indexWriter.addDocument(textDocument);
// BREAKS HERE!!!!
}
} catch (FileNotFoundException fnfe) {
System.out.println(fnfe.getMessage());
return false;
} catch (CorruptIndexException cie) {
throw new CIDBException("The index has become
corrupt.");
} catch (IOException ioe) {
System.out.println(ioe.getMessage());
return false;
}
return true;
}
Thanks much,
Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org