Limit indexed documents.

["tomas.kalas" <ka...@email.cz>]

Hello i have a few questions for indexing data.
Existing some hardware or software limits for indexing data?
And is some maximum of indexed documents?
Thanks for your answers.



--
View this message in context: http://lucene.472066.n3.nabble.com/Limit-indexed-documents-tp4212913.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Limit indexed documents.

[Toke Eskildsen <te...@statsbiblioteket.dk>]

tomas.kalas <ka...@email.cz> wrote:
> Existing some hardware or software limits for indexing data?

The only really hard Solr limit is 2 billion X per shard, where X is document count, unique values in a DocValues String field and other things like that. There are some softer limits, after which performance degrades markedly: Number of fields (hundreds are fine, millions are unrealistic), number of shards (avoid going into the thousands). Having a Java heap of hundreds of gigabytes is possible, but requires tweaking to avoid very long garbage collection pauses. I do not know of a byte size limit for shards: Shards of 1-2 TB works without problems on fitting hardware.

> And is some maximum of indexed documents?

While the limit is 2 billion per single shard, SolrCloud does not have this limitation. A soft limit before doing some custom multi-level setup would thus be around 2000 billion documents, divided across 1000 shards.

- Toke Eskildsen