You are viewing a plain text version of this content. The canonical link for it is here.

Posted to java-user@lucene.apache.org by Sergio <sh...@yahoo.es> on 2004/07/20 19:19:19 UTC

join two indexes

Hi,
i want to join two lucene indexes but i dont know how to do that.

For example i have a student index and a school index. 
In the scholl index i have the studentId field.


How to do that ?
Any idea will be wellcomed.
Thx, Sergio.

Re: Very slow IndexReader.open() performance

Posted by Doug Cutting <cu...@apache.org>.

Optimization should not require huge amounts of memory.  Can you tell a 
bit more about your configuration:  What JVM?  What OS?  How many 
fields?  What mergeFactor have you used?

Also, please attach the output of 'ls -l' of your index directory, as 
well as the stack trace you see when OutOfMemory is thrown.

Thanks,

Doug

Mark Florence wrote:
> Hi -- We have a large index (~4m documents, ~14gb) that we haven't been
> able to optimize for some time, because the JVM throws OutOfMemory, after
> climbing to the maximum we can throw at it, 2gb. 
> 
> In fact, the OutOfMemory condition occurred most recently during a segment 
> merge operation. maxMergeDocs was set to the default, and we seem to have
> gotten around this problem by setting it to some lower value, currently
> 100,000. The index is highly interactive so I took the hint from earlier
> posts to set it to this value.
> 
> Good news! No more OutOfMemory conditions.
> 
> Bad news: now, calling IndexReader.open() is taking 20+ seconds, and it 
> is killing performance.
> 
> I followed the design pattern in another earlier post from Doug. I take a
> batch of deletes, open an IndexReader, perform the deletes, then close it.
> Then I take a batch of adds, open an IndexWriter, perform the adds, then
> close it. Then I get a new IndexSearcher for searching.
> 
> But because the index is so interactive, this sequence repeats itself all
> the time. 
> 
> My question is, is there a better way? Performance was fine when I could
> optimize. Can I hold onto singleton a IndexReader/IndexWriter/IndexSearcher
> to avoid the overhead of the open?
> 
> Any help would be most gratefully received.
> 
> Mark Florence, CTO, AIRS
> mflorence@airsmail.com
> 800-897-7714x1703
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org

Very slow IndexReader.open() performance

Posted by Mark Florence <mf...@airsmail.com>.

Hi -- We have a large index (~4m documents, ~14gb) that we haven't been
able to optimize for some time, because the JVM throws OutOfMemory, after
climbing to the maximum we can throw at it, 2gb. 

In fact, the OutOfMemory condition occurred most recently during a segment 
merge operation. maxMergeDocs was set to the default, and we seem to have
gotten around this problem by setting it to some lower value, currently
100,000. The index is highly interactive so I took the hint from earlier
posts to set it to this value.

Good news! No more OutOfMemory conditions.

Bad news: now, calling IndexReader.open() is taking 20+ seconds, and it 
is killing performance.

I followed the design pattern in another earlier post from Doug. I take a
batch of deletes, open an IndexReader, perform the deletes, then close it.
Then I take a batch of adds, open an IndexWriter, perform the adds, then
close it. Then I get a new IndexSearcher for searching.

But because the index is so interactive, this sequence repeats itself all
the time. 

My question is, is there a better way? Performance was fine when I could
optimize. Can I hold onto singleton a IndexReader/IndexWriter/IndexSearcher
to avoid the overhead of the open?

Any help would be most gratefully received.

Mark Florence, CTO, AIRS
mflorence@airsmail.com
800-897-7714x1703


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org

Here is how to search multiple indexes

Posted by Don Vaillancourt <do...@webimpact.com>.

Here is the code that I use to do multi-index searches:

// create a multi index searcher
IndexSearcher indexes[] = new IndexSearcher[n];  // where n is the number 
of indexes to search

for (int i = 0; i < n; i++)
{
         // use whichever IndexSearcher constructor you want
         // blah is the appropriate value to pass
         indexes[i] = new IndexSearcher(blah);
}

// This is the part which allows you to search multiple indexes
Searcher searcher = new MultiSearcher(indexes);

// do the search
Analyzer analyzer = new StandardAnalyzer();
Query query = QueryParser.parse(expression, colSearch, analyzer);

searcher.search(query);

At 01:19 PM 20/07/2004, you wrote:
>Hi,
>i want to join two lucene indexes but i dont know how to do that.
>
>For example i have a student index and a school index.
>In the scholl index i have the studentId field.
>
>
>How to do that ?
>Any idea will be wellcomed.
>Thx, Sergio.

Don Vaillancourt
Director of Software Development

WEB IMPACT INC.
416-815-2000 ext. 245
email: donv@web-impact.com
web: http://www.web-impact.com

This email message is intended only for the addressee(s)
and contains information that may be confidential and/or
copyright.  If you are not the intended recipient please
notify the sender by reply email and immediately delete
this email. Use, disclosure or reproduction of this email
by anyone other than the intended recipient(s) is strictly
prohibited. No representation is made that this email or
any attachments are free of viruses. Virus scanning is
recommended and is the responsibility of the recipient.

Re: join two indexes

Posted by Daniel Naber <da...@t-online.de>.

On Tuesday 20 July 2004 19:19, Sergio wrote:

> i want to join two lucene indexes but i dont know how to do that.

There are two "addIndexes" methods in IndexWriter which you can use to 
write your own small merge tool (a ready-to-use tool for index merging 
doesn't exist AFAIK).

Regards
 Daniel

-- 
http://www.danielnaber.de

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org