You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Michael Chan <da...@gmail.com> on 2006/05/23 08:26:36 UTC
Running 20mil queries against an index
Hi,
I'm trying to run 20mil+ queries against an index containing 2mil
documents, and it has been quite slow. I've been reading about
MemoryIndex, but it is only a single-document index. As I have quite a
bit of RAM (~20gb), is there a way I could store the index in RAM or
any other way that makes use of it to improve performance?
Thanks.
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Running 20mil queries against an index
Posted by Michael Chan <da...@gmail.com>.
Thanks for that. Does anyone know how much RAM a 5gb index will need?
With mx set to 27gb, it crashes when it flushes buffer at one point.
"bash-2.03$ Exception in thread "main" java.lang.ExceptionInInitializerError
at TaxonomyFinder.RelatedCatsFinder.<init>(RelatedCatsFinder.java:46)
at wikipedia.WikipediaAnalyser$ExtractAbstractHandler.endElement(WikipediaAnalyser.java:295)
at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown
Source)
at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanEndElement(Unknown
Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown
Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown
Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at wikipedia.WikipediaAnalyser.parseAbstracts(WikipediaAnalyser.java:184)
at wikipedia.WikipediaAnalyser.getRelatedCategories(WikipediaAnalyser.java:127)
at TaxonomyFinder.TaxonomyTreeMaker.main(TaxonomyTreeMaker.java:492)
Caused by: java.lang.ArrayIndexOutOfBoundsException: -2097152
at java.util.Vector.elementAt(Unknown Source)
at org.apache.lucene.store.RAMOutputStream.flushBuffer(RAMOutputStream.java:82)
at org.apache.lucene.store.BufferedIndexOutput.flush(BufferedIndexOutput.java:84)
at org.apache.lucene.store.BufferedIndexOutput.writeBytes(BufferedIndexOutput.java:52)
at org.apache.lucene.store.RAMDirectory.<init>(RAMDirectory.java:68)
at org.apache.lucene.store.RAMDirectory.<init>(RAMDirectory.java:95)
at word_coocurrence.WordCooccurrenceFinder.<clinit>(WordCooccurrenceFinder.java:50)
... 13 more"
I build the RAMDirectory from a FSDirectory, using the following:
try {
m_searcher = new IndexSearcher(new RAMDirectory("index"));
} catch (IOException e) {
e.printStackTrace();
System.exit(1);
}
where "index" is the path for the index.
Any help will be much appreciated.
Michael
On 5/23/06, Daniel Naber <lu...@danielnaber.de> wrote:
> On Dienstag 23 Mai 2006 08:26, Michael Chan wrote:
>
> > As I have quite a
> > bit of RAM (~20gb), is there a way I could store the index in RAM or
> > any other way that makes use of it to improve performance?
>
> RAMDirectory has just been fixed (in SVN) to work with indexes > 2 GB.
>
> Regards
> Daniel
>
> --
> http://www.danielnaber.de
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Running 20mil queries against an index
Posted by Daniel Naber <lu...@danielnaber.de>.
On Dienstag 23 Mai 2006 08:26, Michael Chan wrote:
> As I have quite a
> bit of RAM (~20gb), is there a way I could store the index in RAM or
> any other way that makes use of it to improve performance?
RAMDirectory has just been fixed (in SVN) to work with indexes > 2 GB.
Regards
Daniel
--
http://www.danielnaber.de
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Running 20mil queries against an index
Posted by Michael Chan <da...@gmail.com>.
I think I've fixed the problem by changing/fixing RAMOutputStream.java.
On 5/23/06, Muralidharan V <mu...@gmail.com> wrote:
> On 5/23/06, Michael Chan <da...@gmail.com> wrote:
> >
> > As I have quite a bit of RAM (~20gb)
>
>
> And I once had a 486 with 2MB RAM, which was later 'upgraded' to 4MB :-)
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Running 20mil queries against an index
Posted by Muralidharan V <mu...@gmail.com>.
On 5/23/06, Michael Chan <da...@gmail.com> wrote:
>
> As I have quite a bit of RAM (~20gb)
And I once had a 486 with 2MB RAM, which was later 'upgraded' to 4MB :-)