Stress/scalability testing Lucene

[Otis Gospodnetic <ot...@yahoo.com>]

Hello,

Has anyone tested Lucene for scalability?
I know that some peple have indices with 10M+ documents in it, but has
anyone tried going beyond there, to 50M, 100M, 500M or more documents?
(I know the size of the index and performance of searches depends on
documents, number of fields, field types, query complexity, etc.)

Last night I wrote a simple class that creates a Lucene index of
specified size with documents containing 2 fields, one Text with about
24 bytes, and one UnStored without about 16000 bytes.
It took about 8 hours to index 100K documents, resulting in an index of
578 MB (optimized).  This was on 400MHz machine with about 384MB RAM,
doing nothing else.

I then realized that I can't build a relaly big index to test Lucene's
scalability properly, simply because I don't have a big enough disk :)

So my question is:
Has anyone done this type of testing and can you share the results?
Does anyone have a machine with sufficient amount of RAM and disk and
wants to do this?

Thanks,
Otis
P.S.
If anyone is wondering about those 8 hours - this was with a plain
IndexWriter and mergeFactor set to 1000, and java -Xms50M and -Xmx80MB



__________________________________________________
Do you Yahoo!?
Yahoo! Web Hosting - Let the expert host your site
http://webhosting.yahoo.com

--
To unsubscribe, e-mail:   <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>