Solr cloud planning

[gatanathoa <ms...@microfocus.com>]

I am trying to find what setup will give us the fastest searching. But I have
not been able to find out this information with my internet searching.

I have the possibility to have up to 16 identical servers for the index,
what setup will get me the fastest searching?
e.g. 8 shards, 8 replicas, and 15 zookeepers. 
4 shards, 8 replicas, and 5 zookeepers. 
Or any other combination up to the 16 servers.




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr cloud planning

[Erick Erickson <er...@gmail.com>]

You'll almost certainly have to shard then. First of all Lucene has a
hard limit of 2^31 docs in a single index so there's a 2B limit.
There's no such limit on the number of docs in the collection (i.e. 5
shards each can have 2B docs for 10B docs total in the collection).

But nobody that I know of has that many documents on a shard, although
I've seen 200M-300M docs on a shard give good response time. I've also
seen 20M docs strain a beefy server.

Here's an outline of what it takes to find out:
https://lucidworks.com/2012/07/23/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/

The idea is to set up a test environment that you strain to breaking
with _your_ data/queries/environment. You can do this with just two
machines, from there its just multiplying...

Best,
Erick

On Wed, Oct 4, 2017 at 6:07 AM, gatanathoa <ms...@microfocus.com> wrote:
> There is a very large amount of data and there will be a constant addition of
> more data. There will be hundreds of millions if not billions of items.
>
> We have to be able to be able to be constantly indexing items but also allow
> for searching. Sadly there is no way to know the amount of searching that
> will be done, but was told to expected a fair amount. (I have no idea what
> "a fair amount" means either)
>
> I am not sure that only one shard will be adequate in this setup. The speed
> of the search results is the key here. There is also no way to test this
> prior to implementation.
>
> Is this enough information to be able to provide some guide lines?
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr cloud planning

[gatanathoa <ms...@microfocus.com>]

There is a very large amount of data and there will be a constant addition of
more data. There will be hundreds of millions if not billions of items.

We have to be able to be able to be constantly indexing items but also allow
for searching. Sadly there is no way to know the amount of searching that
will be done, but was told to expected a fair amount. (I have no idea what
"a fair amount" means either)

I am not sure that only one shard will be adequate in this setup. The speed
of the search results is the key here. There is also no way to test this
prior to implementation. 

Is this enough information to be able to provide some guide lines?



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr cloud planning

[Shawn Heisey <ap...@elyograg.org>]

On 10/3/2017 3:18 PM, gatanathoa wrote:
> I am trying to find what setup will give us the fastest searching. But I have
> not been able to find out this information with my internet searching.
>
> I have the possibility to have up to 16 identical servers for the index,
> what setup will get me the fastest searching?
> e.g. 8 shards, 8 replicas, and 15 zookeepers.
> 4 shards, 8 replicas, and 5 zookeepers.
> Or any other combination up to the 16 servers.

This question is impossible to answer with the information provided.  
The query rate will have a huge impact on the answer. The types of 
queries will matter.  The size of the index, both in document count and 
physical size on disk, will matter.  Full-scale testing is often the 
only way to be sure.  You can test at a smaller scale, but that may 
behave very differently than the full-size index handling a full 
production load.

Generally speaking, adding shards will only help performance if your 
query rate is very low, because the subqueries to each shard can take 
advantage of a lot of idle CPU resources in that situation.  The more 
queries you have to support per second, the fewer shards you're going to 
want, and that might mean going all the way down to one shard.  If the 
rows and/or start parameters on the requests are very large, you 
probably don't want more than one shard.

Thanks,
Shawn

Re: Solr cloud planning

[John Bickerstaff <jo...@johnbickerstaff.com>]

Others will comment I'm sure, but I'm suspicious the answer is that you can
only know by testing.

The kind and quantity of data in your indexes - the type of queries you
normally receive, etc, etc...

All these things will have an affect on your speed.

For sure you don't want 15 Zookeepers.  That will leave you with only one
dedicated Solr server.

In my experience, for production, you don't want Zookeeper and Solr on the
same box.

If others have different experience, I'm sure they'll share their opinions
too.

I'd recommend starting with 3 Zookeepers (the minimum) on dedicated servers
(Zookeeper only)

Start with basic sharding (like what you mention -- 4 shards, 8 replicas)
and then load test.  You may find that some shards get much more traffic
than others do and in that case you'll want more replicas for those
shards...   It all depends on the patterns of use and the amount of data.



On Tue, Oct 3, 2017 at 3:18 PM, gatanathoa <ms...@microfocus.com>
wrote:

> I am trying to find what setup will give us the fastest searching. But I
> have
> not been able to find out this information with my internet searching.
>
> I have the possibility to have up to 16 identical servers for the index,
> what setup will get me the fastest searching?
> e.g. 8 shards, 8 replicas, and 15 zookeepers.
> 4 shards, 8 replicas, and 5 zookeepers.
> Or any other combination up to the 16 servers.
>
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>