Is there a maximum size for a reader?

[Liz Sommers <li...@gmail.com>]

Lucene 4.2.1

I am moving my application from 3.6 to 4.2.1.  At the moment I have a
reader for each of my shards (100 shards each about 5 million records).  I
am considering using a MultiReader for all the shards.  The application
gets about 2 million new records a week.  Right now I don't have a full
index to test on.

My question is - approximately how many records can a Reader handle?  Does
anybody know about memory constraints or size constraints?  Do to space
constraints I can't build a full index for testing until I have something
that I can nearly guarantee will work.

Thanks
Liz

RE: Is there a maximum size for a reader?

[Uwe Schindler <uw...@thetaphi.de>]

A MultiReader is only limited by the Integer.MAX_VALUE limitation for the number of documents (because IndexReader.document() only uses int as doc-id). MultiReader has no logic at all, it is just a "container" of other index readers. The search runs on the leaves of the IndexReader (IndexReader.leaves()), which are AtomicReaders. The number of AtomicReaders on the whole reader structure (MultiReader with n DirectoryReaders each consisting of m AtomicReaders) and the number of documents per atomic reader is more the limiting factor.

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Liz Sommers [mailto:lizsworks@gmail.com]
> Sent: Friday, April 19, 2013 6:15 PM
> To: java-user@lucene.apache.org
> Subject: Is there a maximum size for a reader?
> 
> Lucene 4.2.1
> 
> I am moving my application from 3.6 to 4.2.1.  At the moment I have a reader
> for each of my shards (100 shards each about 5 million records).  I am
> considering using a MultiReader for all the shards.  The application gets about
> 2 million new records a week.  Right now I don't have a full index to test on.
> 
> My question is - approximately how many records can a Reader handle?
> Does anybody know about memory constraints or size constraints?  Do to
> space constraints I can't build a full index for testing until I have something
> that I can nearly guarantee will work.
> 
> Thanks
> Liz


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org