You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by andynuss <an...@yahoo.com> on 2010/08/14 03:24:41 UTC
scalability limit in terms of numbers of large documents
Hi,
Lets say that I am indexing large book documents broken into chapters. A
typical book that you buy at amazon. What would be the approximate limit to
the number of books that can be indexed slowly and searched quickly. The
search unit would be a chapter, so assume that a book is divided into 15-50
chapters. Any ideas?
Andy
--
View this message in context: http://lucene.472066.n3.nabble.com/scalability-limit-in-terms-of-numbers-of-large-documents-tp1142517p1142517.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: scalability limit in terms of numbers of large documents
Posted by Erick Erickson <er...@gmail.com>.
Here's the Wiki:
http://wiki.apache.org/solr/
You can be reading the eBook in 5 minutes, or order the physical book at:
https://www.packtpub.com/solr-1-4-enterprise-search-server/book?utm_source=lucidimagination.com&utm_medium=bookrev&utm_content=other&utm_campaign=mdb_000374
Lucid Imaginations maintains quite a bit of free information in addition to
their commercial services, try:
http://www.lucidimagination.com
Best
Erick
On Sat, Aug 14, 2010 at 9:16 AM, andynuss <an...@yahoo.com> wrote:
>
> Hi Erick,
>
> My documents are roughly a 0.5 to 1 million chars divide into normal words,
> and divided into 50 chapters, each chapter streamed into a docid unit. So
> a
> search hit is a chapter.
>
> How do I find out more about sharding and SOLR?
>
> Andy
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/scalability-limit-in-terms-of-numbers-of-large-documents-tp1142517p1146449.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: scalability limit in terms of numbers of large documents
Posted by andynuss <an...@yahoo.com>.
Hi Erick,
My documents are roughly a 0.5 to 1 million chars divide into normal words,
and divided into 50 chapters, each chapter streamed into a docid unit. So a
search hit is a chapter.
How do I find out more about sharding and SOLR?
Andy
--
View this message in context: http://lucene.472066.n3.nabble.com/scalability-limit-in-terms-of-numbers-of-large-documents-tp1142517p1146449.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: scalability limit in terms of numbers of large documents
Posted by Erick Erickson <er...@gmail.com>.
As asked, that's really an unanswerable question. The math is pretty easy
in terms of running out of document IDs, but "searched quickly" depends
on too many variables.
I suspect, though, that long before you ran out of document IDs, you'd need
to shard your index, Have you looked at SOLR?
Best
Erick
On Fri, Aug 13, 2010 at 9:24 PM, andynuss <an...@yahoo.com> wrote:
>
> Hi,
>
> Lets say that I am indexing large book documents broken into chapters. A
> typical book that you buy at amazon. What would be the approximate limit
> to
> the number of books that can be indexed slowly and searched quickly. The
> search unit would be a chapter, so assume that a book is divided into 15-50
> chapters. Any ideas?
>
> Andy
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/scalability-limit-in-terms-of-numbers-of-large-documents-tp1142517p1142517.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: scalability limit in terms of numbers of large documents
Posted by "Burton-West, Tom" <tb...@umich.edu>.
Hi Andy,
We are currently indexing about 650,000 full-text books in per Solr/Lucene index. We have 10 shards for a total of about 6.5 million documents and our average response time is under a 2 seconds, but the slowest 1% of queries take between 5-30 seconds. If you were searching only one index of 650,000 documents instead of the 6.5 million, the response time would quite a bit better. If you only allow boolean "AND" queries and use stopwords, the response time would be significantly better. Our slowest searches are almost all phrase queries with common words.
You probably need to define what you mean by "searched quickly" and what kind of a load you are expecting. Also you need to think about what kind of hardware you want to use. Also as index sizes get large,disk I/O can become a bottleneck. Using more memory for the OS disk cache and Solr/Lucene caches can compensate for this. Using SSD's instead of Hard Disks can also offset this as Toke can tell you about. If you need to do frequent index updates it can invalidate both the OS I/O cache and the Solr/Lucene caches, so there are lots of trade-offs to tune.
Lucene had a limit of about 2.4 billion unique terms per segment, which we ran into because we have dirty OCR and 200 languages (http://www.hathitrust.org/blogs/large-scale-search/too-many-words). However Michael McCandless changed the limit to about 274 billion unique terms. Chances are you will run into bottlenecks with disk I/O or other bottlenecks, long before you reach this limit.
BTW: we index whole books as Solr documents, not chapters or pages.
Tom
www.hathitrust.org/blogs
________________________________________
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: scalability limit in terms of numbers of large documents
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Sat, 2010-08-14 at 03:24 +0200, andynuss wrote:
> Lets say that I am indexing large book documents broken into chapters. A
> typical book that you buy at amazon. What would be the approximate limit to
> the number of books that can be indexed slowly and searched quickly. The
> search unit would be a chapter, so assume that a book is divided into 15-50
> chapters. Any ideas?
Hathi Trust has an excellent blog where they write about indexing
5 million+ scanned books. http://www.hathitrust.org/blogs
They focus on OCR'ed books where dirty data is a big problem, but most
of their thoughts and solutions can be used for clean data too.
Regards,
Toke Eskildsen
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org