You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by "Kevin A. Burton" <bu...@newsmonster.org> on 2005/02/15 20:08:22 UTC
Re: Opening up one large index takes 940M or memory?
Doug Cutting wrote:
> Kevin A. Burton wrote:
>
>> Is there any way to reduce this footprint? The index is fully
>> optimized... I'm willing to take a performance hit if necessary. Is
>> this documented anywhere?
>
>
> You can increase TermInfosWriter.indexInterval. You'll need to
> re-write the .tii file for this to take effect. The simplest way to
> do this is to use IndexWriter.addIndexes(), adding your index to a
> new, empty, directory. This will of course take a while for a 60GB
> index...
>
(Note... when this works I'll note my findings in a wiki page for future
developers)
Two more questions:
1. Do I have to do this with a NEW directory? Our nightly index merger
uses an existing "target" index which I assume will re-use the same
settings as before? I did this last night and it still seems to use the
same amount of memory. Above you assert that I should use a new empty
directory and I'll try that tonight.
2. This isn't destructive is it? I mean I'll be able to move BACK to a
TermInfosWriter.indexInterval of 128 right?
Thanks!
Kevin
--
Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an
invite! Also see irc.freenode.net #rojo if you want to chat.
Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html
If you're interested in RSS, Weblogs, Social Networking, etc... then you
should work for Rojo! If you recommend someone and we hire them you'll
get a free iPod!
Kevin A. Burton, Location - San Francisco, CA
AIM/YIM - sfburtonator, Web - http://peerfear.org/
GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Opening up one large index takes 940M or memory?
Posted by Doug Cutting <cu...@apache.org>.
Kevin A. Burton wrote:
> 1. Do I have to do this with a NEW directory? Our nightly index merger
> uses an existing "target" index which I assume will re-use the same
> settings as before? I did this last night and it still seems to use the
> same amount of memory. Above you assert that I should use a new empty
> directory and I'll try that tonight.
You need to re-write the entire index using a modified
TermIndexWriter.java. Optimize rewrites the entire index but is
destructive. Merging into a new empty directory is a non-destructive
way to do this.
> 2. This isn't destructive is it? I mean I'll be able to move BACK to a
> TermInfosWriter.indexInterval of 128 right?
Yes, you can go back if you re-optimize or re-merge again.
Also, there's no need to CC my personal email address.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org