You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by search engn dev <sa...@gmail.com> on 2014/07/18 08:17:28 UTC
SolrCloud performance issues regarding hardware configuration
I need help regarding hardware configuration of my SolrCloud setup. most of
the queries to our SolrCloud are boolean queries involving 6-7 fields and
spatial and facet queries. currently for facet queries solr throws OOM
errors.
below are my solrcloud stats,
solr version 4.6.1
1 collection : 4 shards : each shard has one master and one replica
total documents : 700 million
total index size is 150 GB, i.e 37.5GB at each shard
total fields in index are 30, 25 fields are stored and indexed, remaining
are only indexed.
hardware of each solr node is quad-core cpu and 16 gb ram
JVM parameters : -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Xms8192M
-Xmx8192M
Now what we planned is instead of 4 shards we will create 8 shards of 32GB
ram and quad-core cpu, then each shard will have ~19GB of index.
What will be correct approach here, creating 4 shards with 64 gb ram each or
creating 8 shards with 32 gb ram each?
--
View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-performance-issues-regarding-hardware-configuration-tp4147843.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: SolrCloud performance issues regarding hardware configuration
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
search engn dev [sachinyadav0025@gmail.com] wrote:
> out of 700 million documents 95-97% values are unique approx.
That's quite a lot. If you are not already using DocValues for that, you should do so.
So, each shard handles ~175M documents. Even with DocValues, there is an overhead of just having the base facet structure in memory, but that should be doable with 8GB of heap. It helps if your field is single-value and marked as such in the schema. If your indexes are rarely updated, you can consider optimizing down to a single segment: Single-value-field + DocValues + single-segment = very low memory overhead for the base facet structures.
However, As Eric states, each call temporarily allocates ~175M buckets aka ints, which means 700MB of heap allocation. When you get the call to work, you should limit the number of concurrent searches severely. Remember to do it outside of the Solr Cloud, to avoid dead-locks.
Performance of high-cardinality faceting with stock Solr is not great (but not that bad either, considering what it does), but if you really need it you can work around some of the issues. SOLR-5894 (https://issues.apache.org/jira/browse/SOLR-5894 - I am the author) makes it possible to get the results faster for small result sets and to limit the amount of memory used by the buckets: If the maximum possible count for any term in your user_digest field is 20 (just an example), the buckets need only be 6 bits each (2^6=32 > 20) and the temporary memory allocation for each call will be 175M * 6bit / 8 bits/byte = 132MByte. Read http://sbdevel.wordpress.com/2014/04/04/sparse-facet-counting-without-the-downsides/ for a detailed description.
- Toke Eskildsen
Re: SolrCloud performance issues regarding hardware configuration
Posted by Himanshu Mehrotra <hi...@snapdeal.com>.
Hi,
Increasing the number of replicas per shard will help you take more
concurrent users/queries resulting in increased throughput.
Thanks,
Himanshu
On Mon, Jul 21, 2014 at 9:25 AM, search engn dev <sa...@gmail.com>
wrote:
> Thanks Erick,
>
> /"So your choices are either to increase memory (a lot) or not do this.
> It's a valid question whether this is useful information to present to a
> user
> (or are you doing some kind of analytics here?). "/
>
> Yes, You are right my facet queries are for text analytic purpose. Users
> will send boolean and spatial queries. current performance for spatial
> queries is 100qps with 150 concurrent users and avg response time is 500ms.
> In this case to increase the throughput which is the correct way to go from
> below two.
> 1. Increasing replica of each shard
> 2. Increasing total number shards
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/SolrCloud-performance-issues-regarding-hardware-configuration-tp4147843p4148222.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
--
Himanshu Mehrotra
Download Our App[image: A]
<https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>[image:
A]
<https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>[image:
W]
<http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
*Ext*: 529
246 OKHLA PHASE III, NEW DELHI 110 020, INDIA
[image: Snapdeal.com] <http://www.snapdeal.com/>
RE: SolrCloud performance issues regarding hardware configuration
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
search engn dev [sachinyadav0025@gmail.com] wrote:
> Yes, You are right my facet queries are for text analytic purpose.
Does this mean that facet calls are rare (at most one at a time)?
> Users will send boolean and spatial queries. current performance for spatial
> queries is 100qps with 150 concurrent users and avg response time is 500ms.
What is the limiting factor here? CPU or I/O? If it is the latter, then adding more memory to the existing setup seems like the cheapest and easiest choice.
- Toke Eskildsen
Re: SolrCloud performance issues regarding hardware configuration
Posted by search engn dev <sa...@gmail.com>.
Thanks Erick,
/"So your choices are either to increase memory (a lot) or not do this.
It's a valid question whether this is useful information to present to a
user
(or are you doing some kind of analytics here?). "/
Yes, You are right my facet queries are for text analytic purpose. Users
will send boolean and spatial queries. current performance for spatial
queries is 100qps with 150 concurrent users and avg response time is 500ms.
In this case to increase the throughput which is the correct way to go from
below two.
1. Increasing replica of each shard
2. Increasing total number shards
--
View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-performance-issues-regarding-hardware-configuration-tp4147843p4148222.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: SolrCloud performance issues regarding hardware configuration
Posted by Erick Erickson <er...@gmail.com>.
Right, this is the worst kind of use-case for faceting. You have
150M docs/shard and are asking up to 125M buckets to count
into, plus control structures. Performance of this (even without OOMs)
will be a problem. Having multiple queries execute this simultaneously
will increase memory usage.
So your choices are either to increase memory (a lot) or not do this.
It's a valid question whether this is useful information to present to a
user
(or are you doing some kind of analytics here?).
This feels like an XY problem. What is the use-case you're trying to
support with this?
Best,
Erick
On Fri, Jul 18, 2014 at 5:04 AM, search engn dev <sa...@gmail.com>
wrote:
> out of 700 million documents 95-97% values are unique approx.
>
> My facet query is :
>
> http://localhost:8983/solr/select?q=*:*&rows=0&facet=true&facet.limit=10000&facet.field=user_digest
>
> Above query throws OOM exception as soon as fire it to solr.
>
>
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/SolrCloud-performance-issues-regarding-hardware-configuration-tp4147843p4147871.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
RE: SolrCloud performance issues regarding hardware configuration
Posted by search engn dev <sa...@gmail.com>.
out of 700 million documents 95-97% values are unique approx.
My facet query is :
http://localhost:8983/solr/select?q=*:*&rows=0&facet=true&facet.limit=10000&facet.field=user_digest
Above query throws OOM exception as soon as fire it to solr.
--
View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-performance-issues-regarding-hardware-configuration-tp4147843p4147871.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: SolrCloud performance issues regarding hardware configuration
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
From: search engn dev [sachinyadav0025@gmail.com]:
> 1 collection : 4 shards : each shard has one master and one replica
> total documents : 700 million
Are you using DocValues for your facet fields? What is the approximate number of unique values in your facets and what is their type (string, number...)? Can you give us a sample request that leads to an OOM?
> What will be correct approach here, creating 4 shards with 64 gb ram each or
> creating 8 shards with 32 gb ram each?
Faceting can result in CPU/memory heavy single-threaded calls in Solr when there are a lot of documents. If your main performance problem is latency, go for the 8 shard solution to let more CPUs work at the same time. If you just need to maximize throughput, the difference should not be that big.
- Toke Eskildsen