Question about index sizes.

[Jim Adams <ja...@gmail.com>]

Can anyone give me a rule of thumb for knowing when you need to go to
multicore or shards?  How many records can be in an index before it breaks
down?  Does it break down?  Is it 10 million? 20 million?  50 million?

Thanks, Jim

RE: Question about index sizes.

[Ensdorf Ken <En...@zoominfo.com>]

That's a great question.  And the answer is, of course, it depends.  Mostly on the size of the documents you are indexing.  50 million rows from a database table with a handful of columns is very different from 50 million web pages,  pdf documents, books, etc.

We currently have about 50 million documents split across 2 servers with reasonable performance - sub-second response time in most cases.  The total size of the 2 indices is about 300G.  I'd say most of the size is from stored fields, though we index just about everything.  This is on 64-bit ubuntu boxes with 32G of memory.  We haven't pushed this into production yet, but initial load-testing results look promising.

Hope this helps!

> -----Original Message-----
> From: Jim Adams [mailto:jasolruser@gmail.com]
> Sent: Tuesday, June 23, 2009 1:24 PM
> To: solr-user@lucene.apache.org
> Subject: Question about index sizes.
>
> Can anyone give me a rule of thumb for knowing when you need to go to
> multicore or shards?  How many records can be in an index before it
> breaks
> down?  Does it break down?  Is it 10 million? 20 million?  50 million?
>
> Thanks, Jim