You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by supersoft <el...@gmail.com> on 2011/01/07 10:57:43 UTC

Improving Solr performance

have deployed a 5-sharded infrastructure where: shard1 has 3124422 docs
shard2 has 920414 docs shard3 has 602772 docs shard4 has 2083492 docs shard5
has 11915639 docs Indexes total size: 100GB

The OS is Linux x86_64 (Fedora release 8) with vMem equal to 7872420 and I
run the server using Jetty (from Solr example download) with: java -Xmx3024M
-Dsolr.solr.home=multicore -jar start.jar

The response time for a query is around 2-3 seconds. Nevertheless, if I
execute several queries at the same time the performance goes down
inmediately: 1 simultaneous query: 2516ms 2 simultaneous queries: 4250,4469
ms 3 simultaneous queries: 5781, 6219, 6219 ms 4 simultaneous queries: 6484,
7203, 7719, 7781 ms...

Using JConsole for monitoring the server java proccess I checked that Heap
Memory and the CPU Usages don't reach the upper limits so the server
shouldn't perform as overloaded. Can anyone give me an approach of how I
should tune the instance for not being so hardly dependent of the number of
simultaneous queries?

Thanks in advance
-- 
View this message in context: http://lucene.472066.n3.nabble.com/Improving-Solr-performance-tp2210842p2210842.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Improving Solr performance

Posted by Markus Jelsma <ma...@openindex.io>.
No, it also depends on the queries you execute (sorting is a big consumer) and 
the number of concurrent users.

> Is that a general rule of thumb? That it is best to have about the
> same amount of RAM as the size of your index?
> 
> So, with a 5GB index, I should have between 4GB and 8GB of RAM
> dedicated to solr?

Re: Improving Solr performance

Posted by Markus Jelsma <ma...@openindex.io>.
Any sources to cite for this statement? And are you talking about RAM 
allocated to the JVM or available for OS cache?

> Not sure if this was mentioned yet, but if you are doing slave/master
> replication you'll need 2x the RAM at replication time. Just something to
> keep in mind.
> 
> -mike
> 
> On Mon, Jan 10, 2011 at 5:01 PM, Toke Eskildsen 
<te...@statsbiblioteket.dk>wrote:
> > On Mon, 2011-01-10 at 21:43 +0100, Paul wrote:
> > > > I see from your other messages that these indexes all live on the
> > > > same
> > 
> > machine.
> > 
> > > > You're almost certainly I/O bound, because you don't have enough
> > > > memory
> > 
> > for the
> > 
> > > > OS to cache your index files.  With 100GB of total index size, you'll
> > 
> > get best
> > 
> > > > results with between 64GB and 128GB of total RAM.
> > > 
> > > Is that a general rule of thumb? That it is best to have about the
> > > same amount of RAM as the size of your index?
> > 
> > I does not seems like there is a clear current consensus on hardware to
> > handle IO problems. I am firmly in the SSD camp, but as you can see from
> > the current thread, other people recommend RAM and/or extra machines.
> > 
> > I can say that our tests with RAM and spinning disks showed us that a
> > lot of RAM certainly helps a lot, but also that it takes a considerable
> > amount of time to warm the index before the performance is satisfactory.
> > It might be helped with disk cache tricks, such as copying the whole
> > index to /dev/null before opening it in Solr.
> > 
> > > So, with a 5GB index, I should have between 4GB and 8GB of RAM
> > > dedicated to solr?
> > 
> > Not as -Xmx, but free for disk cache, yes. If you follow the RAM ~=
> > index size recommendation.

Re: Improving Solr performance

Posted by mike anderson <sa...@gmail.com>.
Not sure if this was mentioned yet, but if you are doing slave/master
replication you'll need 2x the RAM at replication time. Just something to
keep in mind.

-mike

On Mon, Jan 10, 2011 at 5:01 PM, Toke Eskildsen <te...@statsbiblioteket.dk>wrote:

> On Mon, 2011-01-10 at 21:43 +0100, Paul wrote:
> > > I see from your other messages that these indexes all live on the same
> machine.
> > > You're almost certainly I/O bound, because you don't have enough memory
> for the
> > > OS to cache your index files.  With 100GB of total index size, you'll
> get best
> > > results with between 64GB and 128GB of total RAM.
> >
> > Is that a general rule of thumb? That it is best to have about the
> > same amount of RAM as the size of your index?
>
> I does not seems like there is a clear current consensus on hardware to
> handle IO problems. I am firmly in the SSD camp, but as you can see from
> the current thread, other people recommend RAM and/or extra machines.
>
> I can say that our tests with RAM and spinning disks showed us that a
> lot of RAM certainly helps a lot, but also that it takes a considerable
> amount of time to warm the index before the performance is satisfactory.
> It might be helped with disk cache tricks, such as copying the whole
> index to /dev/null before opening it in Solr.
>
> > So, with a 5GB index, I should have between 4GB and 8GB of RAM
> > dedicated to solr?
>
> Not as -Xmx, but free for disk cache, yes. If you follow the RAM ~=
> index size recommendation.
>
>

Re: Improving Solr performance

Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Mon, 2011-01-10 at 21:43 +0100, Paul wrote:
> > I see from your other messages that these indexes all live on the same machine.
> > You're almost certainly I/O bound, because you don't have enough memory for the
> > OS to cache your index files.  With 100GB of total index size, you'll get best
> > results with between 64GB and 128GB of total RAM.
> 
> Is that a general rule of thumb? That it is best to have about the
> same amount of RAM as the size of your index?

I does not seems like there is a clear current consensus on hardware to
handle IO problems. I am firmly in the SSD camp, but as you can see from
the current thread, other people recommend RAM and/or extra machines.

I can say that our tests with RAM and spinning disks showed us that a
lot of RAM certainly helps a lot, but also that it takes a considerable
amount of time to warm the index before the performance is satisfactory.
It might be helped with disk cache tricks, such as copying the whole
index to /dev/null before opening it in Solr.

> So, with a 5GB index, I should have between 4GB and 8GB of RAM
> dedicated to solr?

Not as -Xmx, but free for disk cache, yes. If you follow the RAM ~=
index size recommendation.


Re: Improving Solr performance

Posted by Paul <pa...@nines.org>.
> I see from your other messages that these indexes all live on the same machine.
> You're almost certainly I/O bound, because you don't have enough memory for the
> OS to cache your index files.  With 100GB of total index size, you'll get best
> results with between 64GB and 128GB of total RAM.

Is that a general rule of thumb? That it is best to have about the
same amount of RAM as the size of your index?

So, with a 5GB index, I should have between 4GB and 8GB of RAM
dedicated to solr?

Re: Improving Solr performance

Posted by Dennis Gearon <ge...@sbcglobal.net>.
These are definitely server grade machines.

There aren't any desktops I know of (that aren't made for HD video 
editing/rendition) that ever need that kind of memory.

 Dennis Gearon


Signature Warning
----------------
It is always a good idea to learn from your own mistakes. It is usually a better 
idea to learn from others’ mistakes, so you do not have to make them yourself. 
from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'


EARTH has a Right To Life,
otherwise we all die.



----- Original Message ----
From: Shawn Heisey <so...@elyograg.org>
To: solr-user@lucene.apache.org
Sent: Sun, January 9, 2011 4:34:08 PM
Subject: Re: Improving Solr performance

On 1/7/2011 2:57 AM, supersoft wrote:
> have deployed a 5-sharded infrastructure where: shard1 has 3124422 docs
> shard2 has 920414 docs shard3 has 602772 docs shard4 has 2083492 docs shard5
> has 11915639 docs Indexes total size: 100GB
> 
> The OS is Linux x86_64 (Fedora release 8) with vMem equal to 7872420 and I
> run the server using Jetty (from Solr example download) with: java -Xmx3024M
> -Dsolr.solr.home=multicore -jar start.jar
> 
> The response time for a query is around 2-3 seconds. Nevertheless, if I
> execute several queries at the same time the performance goes down
> inmediately: 1 simultaneous query: 2516ms 2 simultaneous queries: 4250,4469
> ms 3 simultaneous queries: 5781, 6219, 6219 ms 4 simultaneous queries: 6484,
> 7203, 7719, 7781 ms...

I see from your other messages that these indexes all live on the same machine.  
You're almost certainly I/O bound, because you don't have enough memory for the 
OS to cache your index files.  With 100GB of total index size, you'll get best 
results with between 64GB and 128GB of total RAM.  Alternatively, you could use 
SSD to store the indexes instead of spinning hard drives, or put each shard on 
its own physical machine with RAM appropriately sized for the index.  For shard5 
on its own machine, at 64GB index size, you might be able to get away with 32GB, 
but ideally you'd want 48-64GB.

Can you do anything to reduce the index size?  Perhaps you are storing fields 
that you don't need to be returned in the search results.  Ideally, you should 
only include enough information to fully populate a search results grid, and 
retrieve detail information for an individual document from the original data 
source instead of Solr.

Thanks,
Shawn

Re: Improving Solr performance

Posted by Shawn Heisey <so...@elyograg.org>.
On 1/7/2011 2:57 AM, supersoft wrote:
> have deployed a 5-sharded infrastructure where: shard1 has 3124422 docs
> shard2 has 920414 docs shard3 has 602772 docs shard4 has 2083492 docs shard5
> has 11915639 docs Indexes total size: 100GB
>
> The OS is Linux x86_64 (Fedora release 8) with vMem equal to 7872420 and I
> run the server using Jetty (from Solr example download) with: java -Xmx3024M
> -Dsolr.solr.home=multicore -jar start.jar
>
> The response time for a query is around 2-3 seconds. Nevertheless, if I
> execute several queries at the same time the performance goes down
> inmediately: 1 simultaneous query: 2516ms 2 simultaneous queries: 4250,4469
> ms 3 simultaneous queries: 5781, 6219, 6219 ms 4 simultaneous queries: 6484,
> 7203, 7719, 7781 ms...

I see from your other messages that these indexes all live on the same 
machine.  You're almost certainly I/O bound, because you don't have 
enough memory for the OS to cache your index files.  With 100GB of total 
index size, you'll get best results with between 64GB and 128GB of total 
RAM.  Alternatively, you could use SSD to store the indexes instead of 
spinning hard drives, or put each shard on its own physical machine with 
RAM appropriately sized for the index.  For shard5 on its own machine, 
at 64GB index size, you might be able to get away with 32GB, but ideally 
you'd want 48-64GB.

Can you do anything to reduce the index size?  Perhaps you are storing 
fields that you don't need to be returned in the search results.  
Ideally, you should only include enough information to fully populate a 
search results grid, and retrieve detail information for an individual 
document from the original data source instead of Solr.

Thanks,
Shawn