About solr distributed search

[Pengkai Qin <qi...@163.com>]

Hi all,

Now I'm doing research on solr distributed search, and it is said documents more than one million is reasonable to use distributed search.
So I want to know, does anyone have the test result(Such as time cost) of using single index and distributed search of more than one million data? I need the test result very urgent, thanks in advance!

Best Regards,
Pengkai

RE: About solr distributed search

["Jaeger, Jay - DOT" <Ja...@dot.wi.gov>]

I am no expert, but here is my take and our situation.

Firstly, are you asking what the minimum number of documents is before it makes *any* sense at all to use a distributed search, or are you asking what the maximum number of documents is before a distributed search is essentially required?  The answers would be different.  I get the feeling you are asking the second question, so I'll proceed under that assumption.

I expect that in part the answer is "it depends".  I expect that it is mostly a function of the size of the index (and the interaction between that and memory and search performance), which depends on both the number of documents and how much is stored for the documents.  It also would depend upon your update load.

If the documents are small and/or the amount of stuff you store per document is small , then until the number of documents and/or updates gets truly enormous a single machine will probably be fine.

But, if your documents (the amount stored per document) is very large, then at some point the index files get so large that performance on a single machine isn't adequate.  Alternatively, if your update load is very very large, you might need to spread out that load among multiple servers to handle the update load without crippling your ability to respond to queries.

As for a specific instance, we have a single index of 7 Million (going on 28 Million), with maybe 512 bytes of data stored for each document, with maybe 26 or so indexed fields (we have a *lot* of copyField operations in order to index the data the way we want it, yet preserve the original data to return), and did not need to use distributed search.

JRJ

-----Original Message-----
From: Pengkai Qin [mailto:qin19890204@163.com] 
Sent: Thursday, September 29, 2011 5:15 AM
To: solr-user@lucene.apache.org; dev@lucene.apache.org
Subject: About solr distributed search

Hi all,

Now I'm doing research on solr distributed search, and it is said documents more than one million is reasonable to use distributed search.
So I want to know, does anyone have the test result(Such as time cost) of using single index and distributed search of more than one million data? I need the test result very urgent, thanks in advance!

Best Regards,
Pengkai

Re: About solr distributed search

[Gregor Kaczor <gk...@gmx.de>]

Hi Pengkai,

my experience is based on http://www.findfiles.net/ which holds >700 Mio 
documents, each about 2kb size.

A single Index containing that kind of data should hold below 80 Mio 
documents. In case you have complex queries with lots of facets, 
sorting, function queries then even 50 Mio documents per index could be 
your upper limit. On very fast Hardware and warmed index you might 
deliver results on average within 1 second.

For documents above 5kb in size those numbers might not necessarily be 
the same.

Try to test your documents by creating (NOT COPYING) and index them in 
vast numbers. After every 10 Mio documents test the average response 
time with caches switched off. If the average response time hits your 
threshold, then the number of documents in index is your limit per index.

Scaling up is no problem. AFAIK 20 to 50 indexes should be fine within a 
distributed productive system.

Kind Regards
Gregor

On 09/29/2011 12:14 PM, Pengkai Qin wrote:
> Hi all,
>
> Now I'm doing research on solr distributed search, and it is said 
> documents more than one million is reasonable to use distributed search.
> So I want to know, does anyone have the test result(Such as time cost) 
> of using single index and distributed search of more than one million 
> data? I need the test result very urgent, thanks in advance!
>
> Best Regards,
> Pengkai
>
>
>
>


-- 
How to find files on the Internet? FindFiles.net <http://findfiles.net>!

Re: About solr distributed search

[Erick Erickson <er...@gmail.com>]

Please don't cross-post to the dev list too, that
list is for internal Solr/Lucene development discussions.

As Jay says, "it depends". The general advice is that
you should use a single index (perhaps replicated)
before you start trying to go the distributed route
(sharding). It's like asking "how big is a Java
program?"...

As for "how many documents", you need to hook up
a stress test process (say jMeter or SolrMeter) and
fire requests at your server at your target speed
to determine whether you have too many documents
on the server or not. And it also depends upon
what your hardware is. As asked, your question
is completely unanswerable.

I've seen 40-50M documents on a single machine.
And that wasn't necessarily a very big one.

Best
Erick

2011/9/29 Pengkai Qin <qi...@163.com>:
> Hi all,
>
> Now I'm doing research on solr distributed search, and it is said documents
> more than one million is reasonable to use distributed search.
> So I want to know, does anyone have the test result(Such as time cost) of
> using single index and distributed search of more than one million data? I
> need the test result very urgent, thanks in advance!
>
> Best Regards,
> Pengkai
>
>
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org