You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by supersoft <el...@gmail.com> on 2011/01/07 10:57:43 UTC
Improving Solr performance
have deployed a 5-sharded infrastructure where: shard1 has 3124422 docs
shard2 has 920414 docs shard3 has 602772 docs shard4 has 2083492 docs shard5
has 11915639 docs Indexes total size: 100GB
The OS is Linux x86_64 (Fedora release 8) with vMem equal to 7872420 and I
run the server using Jetty (from Solr example download) with: java -Xmx3024M
-Dsolr.solr.home=multicore -jar start.jar
The response time for a query is around 2-3 seconds. Nevertheless, if I
execute several queries at the same time the performance goes down
inmediately: 1 simultaneous query: 2516ms 2 simultaneous queries: 4250,4469
ms 3 simultaneous queries: 5781, 6219, 6219 ms 4 simultaneous queries: 6484,
7203, 7719, 7781 ms...
Using JConsole for monitoring the server java proccess I checked that Heap
Memory and the CPU Usages don't reach the upper limits so the server
shouldn't perform as overloaded. Can anyone give me an approach of how I
should tune the instance for not being so hardly dependent of the number of
simultaneous queries?
Thanks in advance
--
View this message in context: http://lucene.472066.n3.nabble.com/Improving-Solr-performance-tp2210842p2210842.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Improving Solr performance
Posted by Markus Jelsma <ma...@openindex.io>.
No, it also depends on the queries you execute (sorting is a big consumer) and
the number of concurrent users.
> Is that a general rule of thumb? That it is best to have about the
> same amount of RAM as the size of your index?
>
> So, with a 5GB index, I should have between 4GB and 8GB of RAM
> dedicated to solr?
Re: Improving Solr performance
Posted by Markus Jelsma <ma...@openindex.io>.
Any sources to cite for this statement? And are you talking about RAM
allocated to the JVM or available for OS cache?
> Not sure if this was mentioned yet, but if you are doing slave/master
> replication you'll need 2x the RAM at replication time. Just something to
> keep in mind.
>
> -mike
>
> On Mon, Jan 10, 2011 at 5:01 PM, Toke Eskildsen
<te...@statsbiblioteket.dk>wrote:
> > On Mon, 2011-01-10 at 21:43 +0100, Paul wrote:
> > > > I see from your other messages that these indexes all live on the
> > > > same
> >
> > machine.
> >
> > > > You're almost certainly I/O bound, because you don't have enough
> > > > memory
> >
> > for the
> >
> > > > OS to cache your index files. With 100GB of total index size, you'll
> >
> > get best
> >
> > > > results with between 64GB and 128GB of total RAM.
> > >
> > > Is that a general rule of thumb? That it is best to have about the
> > > same amount of RAM as the size of your index?
> >
> > I does not seems like there is a clear current consensus on hardware to
> > handle IO problems. I am firmly in the SSD camp, but as you can see from
> > the current thread, other people recommend RAM and/or extra machines.
> >
> > I can say that our tests with RAM and spinning disks showed us that a
> > lot of RAM certainly helps a lot, but also that it takes a considerable
> > amount of time to warm the index before the performance is satisfactory.
> > It might be helped with disk cache tricks, such as copying the whole
> > index to /dev/null before opening it in Solr.
> >
> > > So, with a 5GB index, I should have between 4GB and 8GB of RAM
> > > dedicated to solr?
> >
> > Not as -Xmx, but free for disk cache, yes. If you follow the RAM ~=
> > index size recommendation.
Re: Improving Solr performance
Posted by mike anderson <sa...@gmail.com>.
Not sure if this was mentioned yet, but if you are doing slave/master
replication you'll need 2x the RAM at replication time. Just something to
keep in mind.
-mike
On Mon, Jan 10, 2011 at 5:01 PM, Toke Eskildsen <te...@statsbiblioteket.dk>wrote:
> On Mon, 2011-01-10 at 21:43 +0100, Paul wrote:
> > > I see from your other messages that these indexes all live on the same
> machine.
> > > You're almost certainly I/O bound, because you don't have enough memory
> for the
> > > OS to cache your index files. With 100GB of total index size, you'll
> get best
> > > results with between 64GB and 128GB of total RAM.
> >
> > Is that a general rule of thumb? That it is best to have about the
> > same amount of RAM as the size of your index?
>
> I does not seems like there is a clear current consensus on hardware to
> handle IO problems. I am firmly in the SSD camp, but as you can see from
> the current thread, other people recommend RAM and/or extra machines.
>
> I can say that our tests with RAM and spinning disks showed us that a
> lot of RAM certainly helps a lot, but also that it takes a considerable
> amount of time to warm the index before the performance is satisfactory.
> It might be helped with disk cache tricks, such as copying the whole
> index to /dev/null before opening it in Solr.
>
> > So, with a 5GB index, I should have between 4GB and 8GB of RAM
> > dedicated to solr?
>
> Not as -Xmx, but free for disk cache, yes. If you follow the RAM ~=
> index size recommendation.
>
>
Re: Improving Solr performance
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Mon, 2011-01-10 at 21:43 +0100, Paul wrote:
> > I see from your other messages that these indexes all live on the same machine.
> > You're almost certainly I/O bound, because you don't have enough memory for the
> > OS to cache your index files. With 100GB of total index size, you'll get best
> > results with between 64GB and 128GB of total RAM.
>
> Is that a general rule of thumb? That it is best to have about the
> same amount of RAM as the size of your index?
I does not seems like there is a clear current consensus on hardware to
handle IO problems. I am firmly in the SSD camp, but as you can see from
the current thread, other people recommend RAM and/or extra machines.
I can say that our tests with RAM and spinning disks showed us that a
lot of RAM certainly helps a lot, but also that it takes a considerable
amount of time to warm the index before the performance is satisfactory.
It might be helped with disk cache tricks, such as copying the whole
index to /dev/null before opening it in Solr.
> So, with a 5GB index, I should have between 4GB and 8GB of RAM
> dedicated to solr?
Not as -Xmx, but free for disk cache, yes. If you follow the RAM ~=
index size recommendation.
Re: Improving Solr performance
Posted by Paul <pa...@nines.org>.
> I see from your other messages that these indexes all live on the same machine.
> You're almost certainly I/O bound, because you don't have enough memory for the
> OS to cache your index files. With 100GB of total index size, you'll get best
> results with between 64GB and 128GB of total RAM.
Is that a general rule of thumb? That it is best to have about the
same amount of RAM as the size of your index?
So, with a 5GB index, I should have between 4GB and 8GB of RAM
dedicated to solr?
Re: Improving Solr performance
Posted by Dennis Gearon <ge...@sbcglobal.net>.
These are definitely server grade machines.
There aren't any desktops I know of (that aren't made for HD video
editing/rendition) that ever need that kind of memory.
Dennis Gearon
Signature Warning
----------------
It is always a good idea to learn from your own mistakes. It is usually a better
idea to learn from others’ mistakes, so you do not have to make them yourself.
from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
EARTH has a Right To Life,
otherwise we all die.
----- Original Message ----
From: Shawn Heisey <so...@elyograg.org>
To: solr-user@lucene.apache.org
Sent: Sun, January 9, 2011 4:34:08 PM
Subject: Re: Improving Solr performance
On 1/7/2011 2:57 AM, supersoft wrote:
> have deployed a 5-sharded infrastructure where: shard1 has 3124422 docs
> shard2 has 920414 docs shard3 has 602772 docs shard4 has 2083492 docs shard5
> has 11915639 docs Indexes total size: 100GB
>
> The OS is Linux x86_64 (Fedora release 8) with vMem equal to 7872420 and I
> run the server using Jetty (from Solr example download) with: java -Xmx3024M
> -Dsolr.solr.home=multicore -jar start.jar
>
> The response time for a query is around 2-3 seconds. Nevertheless, if I
> execute several queries at the same time the performance goes down
> inmediately: 1 simultaneous query: 2516ms 2 simultaneous queries: 4250,4469
> ms 3 simultaneous queries: 5781, 6219, 6219 ms 4 simultaneous queries: 6484,
> 7203, 7719, 7781 ms...
I see from your other messages that these indexes all live on the same machine.
You're almost certainly I/O bound, because you don't have enough memory for the
OS to cache your index files. With 100GB of total index size, you'll get best
results with between 64GB and 128GB of total RAM. Alternatively, you could use
SSD to store the indexes instead of spinning hard drives, or put each shard on
its own physical machine with RAM appropriately sized for the index. For shard5
on its own machine, at 64GB index size, you might be able to get away with 32GB,
but ideally you'd want 48-64GB.
Can you do anything to reduce the index size? Perhaps you are storing fields
that you don't need to be returned in the search results. Ideally, you should
only include enough information to fully populate a search results grid, and
retrieve detail information for an individual document from the original data
source instead of Solr.
Thanks,
Shawn
Re: Improving Solr performance
Posted by Shawn Heisey <so...@elyograg.org>.
On 1/7/2011 2:57 AM, supersoft wrote:
> have deployed a 5-sharded infrastructure where: shard1 has 3124422 docs
> shard2 has 920414 docs shard3 has 602772 docs shard4 has 2083492 docs shard5
> has 11915639 docs Indexes total size: 100GB
>
> The OS is Linux x86_64 (Fedora release 8) with vMem equal to 7872420 and I
> run the server using Jetty (from Solr example download) with: java -Xmx3024M
> -Dsolr.solr.home=multicore -jar start.jar
>
> The response time for a query is around 2-3 seconds. Nevertheless, if I
> execute several queries at the same time the performance goes down
> inmediately: 1 simultaneous query: 2516ms 2 simultaneous queries: 4250,4469
> ms 3 simultaneous queries: 5781, 6219, 6219 ms 4 simultaneous queries: 6484,
> 7203, 7719, 7781 ms...
I see from your other messages that these indexes all live on the same
machine. You're almost certainly I/O bound, because you don't have
enough memory for the OS to cache your index files. With 100GB of total
index size, you'll get best results with between 64GB and 128GB of total
RAM. Alternatively, you could use SSD to store the indexes instead of
spinning hard drives, or put each shard on its own physical machine with
RAM appropriately sized for the index. For shard5 on its own machine,
at 64GB index size, you might be able to get away with 32GB, but ideally
you'd want 48-64GB.
Can you do anything to reduce the index size? Perhaps you are storing
fields that you don't need to be returned in the search results.
Ideally, you should only include enough information to fully populate a
search results grid, and retrieve detail information for an individual
document from the original data source instead of Solr.
Thanks,
Shawn