You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Bernd Fehling <be...@uni-bielefeld.de> on 2011/08/02 12:44:17 UTC
performance crossover between single index and sharding
Is there any knowledge on this list about the performance
crossover between a single index and sharding and
when to change from a single index to sharding?
E.g. if index size is larger than 150GB and num of docs is
more than 25 mio. then it is better to change from single index
to sharding and have two shards.
Or something like this...
Sure, solr might even handle 50 mio. docs but performance is going down
and a sharded system with distributed search will be faster than
a single index, or not?
Is a single index always fast than sharding?
Regards
Bernd
Re: performance crossover between single index and sharding
Posted by Peter Keegan <pe...@gmail.com>.
We have 16 shards on 4 physical servers. Shard size was determined by
measuring query response times as a function of doc count. Multiple shards
per server provides parallelism. In a VM environment, I would lean towards 1
shard per VM (with 1/4 the RAM). We implemented our own distributed search
(pre-Solr) and the extra sort/merge processing is not a performance issue.
Peter
On Tue, Aug 2, 2011 at 2:35 PM, Burton-West, Tom <tb...@umich.edu> wrote:
> Hi Jonothan and Markus,
>
> >>Why 3 shards on one machine instead of one larger shard per machine?
>
> Good question!
>
> We made this architectural decision several years ago and I'm not
> remembering the rationale at the moment. I believe we originally made the
> decision due to some tests showing a sweetspot for I/O performance for
> shards with 500,000-600,000 documents, but those tests were made before we
> implemented CommonGrams and when we were still using attached storage. I
> think we also might have had concerns about Java OOM errors with a really
> large shard/index, but we now know that we can keep memory usage under
> control by tweaking the amount of the terms index that gets read into
> memory.
>
> We should probably do some tests and revisit the question.
>
> The reason we don't have 12 shards on 12 machines is that current
> performance is good enough that we can't justify buying 8 more machines:)
>
> Tom
>
> -----Original Message-----
> From: Markus Jelsma [mailto:markus.jelsma@openindex.io]
> Sent: Tuesday, August 02, 2011 2:12 PM
> To: solr-user@lucene.apache.org
> Subject: Re: performance crossover between single index and sharding
>
> Hi Tom,
>
> Very interesting indeed! But i keep wondering why some engineers choose to
> store multiple shards of the same index on the same machine, there must be
> significant overhead. The only reason i can think of is ease of maintenance
> in
> moving shards to a separate physical machine.
> I know that rearranging the shard topology can be a real pain in a large
> existing cluster (e.g. consistent hashing is not consistent anymore and
> having
> to shuffle docs to their new shard), is this the reason you choose this
> approach?
>
> Cheers,
> bble.com.
>
RE: performance crossover between single index and sharding
Posted by "Burton-West, Tom" <tb...@umich.edu>.
Hi Jonothan and Markus,
>>Why 3 shards on one machine instead of one larger shard per machine?
Good question!
We made this architectural decision several years ago and I'm not remembering the rationale at the moment. I believe we originally made the decision due to some tests showing a sweetspot for I/O performance for shards with 500,000-600,000 documents, but those tests were made before we implemented CommonGrams and when we were still using attached storage. I think we also might have had concerns about Java OOM errors with a really large shard/index, but we now know that we can keep memory usage under control by tweaking the amount of the terms index that gets read into memory.
We should probably do some tests and revisit the question.
The reason we don't have 12 shards on 12 machines is that current performance is good enough that we can't justify buying 8 more machines:)
Tom
-----Original Message-----
From: Markus Jelsma [mailto:markus.jelsma@openindex.io]
Sent: Tuesday, August 02, 2011 2:12 PM
To: solr-user@lucene.apache.org
Subject: Re: performance crossover between single index and sharding
Hi Tom,
Very interesting indeed! But i keep wondering why some engineers choose to
store multiple shards of the same index on the same machine, there must be
significant overhead. The only reason i can think of is ease of maintenance in
moving shards to a separate physical machine.
I know that rearranging the shard topology can be a real pain in a large
existing cluster (e.g. consistent hashing is not consistent anymore and having
to shuffle docs to their new shard), is this the reason you choose this
approach?
Cheers,
bble.com.
Re: performance crossover between single index and sharding
Posted by Markus Jelsma <ma...@openindex.io>.
Hi Tom,
Very interesting indeed! But i keep wondering why some engineers choose to
store multiple shards of the same index on the same machine, there must be
significant overhead. The only reason i can think of is ease of maintenance in
moving shards to a separate physical machine.
I know that rearranging the shard topology can be a real pain in a large
existing cluster (e.g. consistent hashing is not consistent anymore and having
to shuffle docs to their new shard), is this the reason you choose this
approach?
Cheers,
> Hi Markus,
>
> Just as a data point for a very large sharded index, we have the full text
> of 9.3 million books with an index size of about 6+ TB spread over 12
> shards on 4 machines. Each machine has 3 shards. The size of each shard
> ranges between 475GB and 550GB. We are definitely I/O bound. Our machines
> have 144GB of memory with about 16GB dedicated to the tomcat instance
> running the 3 Solr instances, which leaves about 120 GB (or 40GB per
> shard) for the OS disk cache. We release a new index every morning and
> then warm the caches with several thousand queries. I probably should add
> that our disk storage is a very high performance Isilon appliance that has
> over 500 drives and every block of every file is striped over no less than
> 14 different drives. (See blog for details *)
>
> We have a very low number of queries per second (0.3-2 qps) and our modest
> response time goal is to keep 99th percentile response time for our
> application (i.e. Solr + application) under 10 seconds.
>
> Our current performance statistics are:
>
> average response time 300 ms
> median response time 113 ms
> 90th percentile 663 ms
> 95th percentile 1,691 ms
>
> We had plans to do some performance testing to determine the optimum shard
> size and optimum number of shards per machine, but that has remained on
> the back burner for a long time as other higher priority items keep
> pushing it down on the todo list.
>
> We would be really interested to hear about the experiences of people who
> have so many shards that the overhead of distributing the queries, and
> consolidating/merging the responses becomes a serious issue.
>
>
> Tom Burton-West
>
> http://www.hathitrust.org/blogs/large-scale-search
>
> *
> http://www.hathitrust.org/blogs/large-scale-search/scaling-large-scale-sea
> rch-500000-volumes-5-million-volumes-and-beyond
>
> -----Original Message-----
> From: Markus Jelsma [mailto:markus.jelsma@openindex.io]
> Sent: Tuesday, August 02, 2011 12:33 PM
> To: solr-user@lucene.apache.org
> Subject: Re: performance crossover between single index and sharding
>
> Actually, i do worry about it. Would be marvelous if someone could provide
> some metrics for an index of many terabytes.
>
> > [..] At some extreme point there will be diminishing
> > returns and a performance decrease, but I wouldn't worry about that at
> > all until you've got many terabytes -- I don't know how many but don't
> > worry about it.
> >
> > ~ David
> >
> > -----
> >
> > Author: https://www.packtpub.com/solr-1-4-enterprise-search-server/book
> >
> > --
> > View this message in context:
> > http://lucene.472066.n3.nabble.com/performance-crossover-between-single-i
> > n dex-and-sharding-tp3218561p3219397.html Sent from the Solr - User
> > mailing list archive at Nabble.com.
Re: performance crossover between single index and sharding
Posted by Dmitry Kan <dm...@gmail.com>.
OK, here is a brief on our sharded setup.
We have 10 shards, 3 per high-end Amazon machine. Majority of the searches
are done on 2 shards at most, that have the latest data in their indices. We
use logical sharding, not hash based. These two lead to a situation, where
given a user query that *will for sure* hit the 2 last (or adjacent in time)
shards, other solr shards would have to search in vain. Therefore, we have
implemented the query router, which is essentially solr itself with
modifications in the QueryComponent. Before implementing the router it was
nearly impossible to run the system.
Why did we do the sharding?
Simply because we started to see a lot OOM exceptions, and various other
instability issues. Also we had to rebuild the index very often due to
changes in the preceeding pipeline. Therefore distributing over shards was
another asset for us in the sense, that reindexing could be carried out in
parallel. On top of that, which is certainly not least, our search became
faster, the slimmer we kept the shards.
We don't yet have master / slave architecture, as this is done when the user
base grows. We started with growing amounts of data, therefore came
horizontal scaling.
Regards,
Dmitry Kan
twitter.com/DmitryKan
On Wed, Aug 3, 2011 at 12:24 PM, Bernd Fehling <
bernd.fehling@uni-bielefeld.de> wrote:
>
> On 02.08.2011 21:00, Shawn Heisey wrote:
>
>> ...
>>
>> I did try some early tests with a single large index. Performance was
>> pretty decent once it got warmed up, but I was worried about how it would
>> perform under a heavy load, and how it would cope with frequent updates. I
>> never really got very far with testing those fears, because the full
>> rebuild time was unacceptable - at least 8 hours. The source database can
>> keep up with six DIH instances reindexing at once, which completes
>> much quicker than a single machine grabbing the entire database. I may
>> increase the number of shards after I remove virtualization, but I'll
>> need to fix a few limitations in my build system.
>> ...
>>
>
> At first, thanks a lot to all answers and here is my setup.
>
> I know that it is very difficult to give specific recommendations about
> this.
> Because of changing from FAST Search to Solr I can state that Solr performs
> very well, if not excellent.
>
> To show that I compare apples and oranges here are my previous FAST Search
> setup:
> - one master server (controlling, logging, search dispatcher)
> - six index server (4.25 mio docs per server, 5 slices per index)
> (searching and indexing at the same time, indexing once per week during
> the weekend)
> - each server has 4GB RAM, all servers are physical on seperate machines
> - RAM usage controlled by the processes
> - total of 25.5 mio. docs (mainly metadata) from 1500 databases worldwide
> - index size is about 67GB per indexer --> about 402GB total
> - about 3 qps at peek times
> - with average search time of 0.05 seconds at peek times
>
> And here is now my current Solr setup:
> - one master server (indexing only)
> - two slave server (search only) but only one is online, the second is
> fallback
> - each server has 32GB RAM, all server are virtuell
> (master on a seperate physical machine, both slaves together on a physical
> machine)
> - RAM usage is currently 20GB to java heap
> - total of 31 mio. docs (all metadata) from 2000 databases worldwide
> - index size is 156GB total
> - search handler statistic report 0.6 average requests per second
> - average time per request 39.5 (is that seconds?)
> - building the index from scratch takes about 20 hours
>
> The good thing is I have the ability to compare a commercial product and
> enterprise system to open source.
>
> I started with my simple Solr setup because of "kiss" (keep it simple and
> stupid).
> Actually it is doing excellent as single index on a single virtuell server.
> But the average time per request should be reduced now, thats why I started
> this discussion.
> While searches with smaller Solr index size (3 mio. docs) showed that it
> can
> stand with FAST Search it now shows that its time to go with sharding.
> I think we are already far behind the point of search performance
> crossover.
>
> What I hope to get with sharding:
> - reduce time for building the index
> - reduce average time per request
>
> What I fear with sharding:
> - i currently have master/slave, do I then have e.g. 3 master and 3 slaves?
> - the query changes because of sharding (is there a search distributor?)
> - how to distribute the content the indexer with DIH on 3 server?
> - anything else to think about while changing to sharding?
>
> Conclusion:
> - Solr can handle much more than 30 mio. docs of metadata in a single index
> if java heap size is large enough. Have an eye on Lucenes fieldCache and
> sorted fields, especially title (string) fields.
> - The crossover in my case is somewhere between 3 mio. and 10 mio. docs
> per index for Solr (compared to FAST Search). FAST recommends about 3 to 6
> mio.
> docs per 4GB RAM server for their system.
>
> Anyone able to reduce my fears about sharding?
> Thanks again for all your answers.
>
> Regards
> Bernd
>
> --
> *****************************************************************
> BASE - Bielefeld Academic Search Engine - www.base-search.net
> *****************************************************************
>
Re: performance crossover between single index and sharding
Posted by Shawn Heisey <so...@elyograg.org>.
On 8/4/2011 12:38 AM, Bernd Fehling wrote:
> Hi Shawn,
>
> the 0.05 seconds for search time at peek times (3 qps) is my target
> for Solr.
> The numbers for solr are from Solr's statistic report page. So 39.5
> seconds
> average per request is definately to long and I have to change to
> sharding.
Solr reports all query times in milliseconds. 39.5 would be 0.0395 seconds.
> For FAST system the numbers for the search dispatcher are:
> 0.042 sec elapsed per normal search, on avg.
> 0.053 sec average uncached normal search time (last 100 queries).
> 99.898% of searches using < 1 sec
> 99.999% of searches using < 3 sec
> 0.000% of all requests timed out
> 22454567.577 sec time up (that is 259 days)
>
> Is there a report page for those numbers for Solr?
The Solr statistics normally page reports averages, but not percentile
statistics. You can add percentile-based statistics (on a limited
subset of your queries) to a 3.X or trunk (4.0) version with SOLR-1972.
I am using this patch in production. Alternatively, you can use INFO
logging in Solr and crawl the logfiles to gather statistics. In the
list below, (the "standard" section on the stats page) the ones that
start with "rolling" are provided by the patch, the others are included
by default. Remember that all these times are in milliseconds.
handlerStart : 1312433464327
requests : 24112
errors : 547
timeouts : 0
totalTime : 2565584
avgTimePerRequest : 106.40279
avgRequestsPerSecond : 0.7097045
rollingRequests : 16384
rollingTotalTime : 1594420
rollingAvgTimePerRequest : 97.315674
rollingAvgRequestsPerSecond : 0.74394274
rollingMedian : 16
rolling75thPercentile : 35
rolling95thPercentile : 225
rolling99thPercentile : 2202
rollingMax : 9397
> About the RAM, the 32GB RAM sind physical for each VM and the 20GB RAM
> are -Xmx for Java.
> Yesterday I noticed that we are running out of heap during replication
> so I have to
> increase -Xmx to about 22g.
That doesn't leave much RAM for the OS disk cache, the primary way to
speed things up with Solr. You should check how long it takes to warm
your caches when you commit, you can find that on the stats page. It's
probably a good idea to lower your autowarmCount values.
If you sharded, you could drop your Java heap size and get more of your
index into RAM. I have a heap size of 3GB for an 18.25GB index (total
of all shards is about 110GB) and do not expect to be increasing that
unless we have problems when we start using faceting, spellchecking, and
suggestions. I have made particular tweaks to garbage collection and
wrote about my experiences on this list. My memory-related java parameters:
-Xms3072M -Xmx3072M
-XX:NewSize=2048M
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
> The reported 0.6 average requests per second seams to me right because
> the Solr system isn't under full load yet. The FAST system is still
> taking
> most of the load. I plan to switch completely to Solr after sharding
> is up and
> running stable. So there will be additional 3 qps to Solr at peek times.
When I read that before, I thought you were saying it was 0.6 seconds
per request, not requests per second. My apologies. A qps of 3 is
quite low. I've seen numbers mentioned here above 30000 qps, and I'm
sure some of the list veterans have seen much higher.
> I don't know if a controlling master like FAST makes any sense for Solr.
> The small VMs with heartbeat and haproxy sounds great, must be on my
> todo list.
If you don't create a core to automatically add the shards parameter
(the master server idea), your application will have to include the
parameter on every request, which means it must be aware of how you have
sharded your index. If that's acceptable to you, there's no problem.
In my case, every single Solr instance has a copy of this broker core.
I only use it on two of them, the two that the load balancer knows about.
> But the biggest problem currently is, how to configure the DIH to
> split up the
> content to several indexer. Is there an indexing distributor?
There is currently no way to have Solr figure out distributed indexing.
Solr doesn't know how you have sharded your data, and it cannot keep
track of primary/secondary indexers. Your build system must figure
these things out. My dih-config.xml accepts variables via the URL,
which I use to tailor my SQL queries.
SELECT * FROM ${dataimporter.request.dataView}
WHERE (
(
did > ${dataimporter.request.minDid}
AND did <= ${dataimporter.request.maxDid}
)
${dataimporter.request.extraWhere}
) AND (crc32(did) % ${dataimporter.request.numShards})
IN (${dataimporter.request.modVal})
I index all new content to a smaller index which I have called the
incremental. The updates that run every two minutes include a modVal
for the above query of "0 1 2 3 4 5". Once a night, I figure out which
content is older than one week. I index that content into the larger
static shards and then delete it from the incremental. This ensures
that commits happen quickly on the index where new content goes. Other
processes that hit all shards (like deletes) run every ten minutes and
check for document presence before they operate, so I will often go an
hour or more between actual updates to the larger indexes, giving the
Solr caches a longer lifetime.
Thanks,
Shawn
Re: performance crossover between single index and sharding
Posted by Bernd Fehling <be...@uni-bielefeld.de>.
java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)
java: file format elf64-x86-64
Including the -d64 switch.
Am 04.08.2011 14:40, schrieb Bob Sandiford:
> Dumb question time - you are using a 64 bit Java, and not a 32 bit Java?
>
> Bob Sandiford | Lead Software Engineer | SirsiDynix
> P: 800.288.8020 X6943 | Bob.Sandiford@sirsidynix.com
> www.sirsidynix.com
>
>
>> -----Original Message-----
>> From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
>> Sent: Thursday, August 04, 2011 2:39 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: performance crossover between single index and sharding
>>
>> Hi Shawn,
>>
>> the 0.05 seconds for search time at peek times (3 qps) is my target for
>> Solr.
>> The numbers for solr are from Solr's statistic report page. So 39.5
>> seconds
>> average per request is definately to long and I have to change to
>> sharding.
>>
>> For FAST system the numbers for the search dispatcher are:
>> 0.042 sec elapsed per normal search, on avg.
>> 0.053 sec average uncached normal search time (last 100 queries).
>> 99.898% of searches using< 1 sec
>> 99.999% of searches using< 3 sec
>> 0.000% of all requests timed out
>> 22454567.577 sec time up (that is 259 days)
>>
>> Is there a report page for those numbers for Solr?
>>
>> About the RAM, the 32GB RAM sind physical for each VM and the 20GB RAM
>> are -Xmx for Java.
>> Yesterday I noticed that we are running out of heap during replication
>> so I have to
>> increase -Xmx to about 22g.
>>
>> The reported 0.6 average requests per second seams to me right because
>> the Solr system isn't under full load yet. The FAST system is still
>> taking
>> most of the load. I plan to switch completely to Solr after sharding is
>> up and
>> running stable. So there will be additional 3 qps to Solr at peek
>> times.
>>
>> I don't know if a controlling master like FAST makes any sense for
>> Solr.
>> The small VMs with heartbeat and haproxy sounds great, must be on my
>> todo list.
>>
>> But the biggest problem currently is, how to configure the DIH to split
>> up the
>> content to several indexer. Is there an indexing distributor?
>>
>> Regards,
>> Bernd
>>
>>
>> Am 03.08.2011 16:33, schrieb Shawn Heisey:
>>> Replies inline.
>>>
>>> On 8/3/2011 2:24 AM, Bernd Fehling wrote:
>>>> To show that I compare apples and oranges here are my previous FAST
>> Search setup:
>>>> - one master server (controlling, logging, search dispatcher)
>>>> - six index server (4.25 mio docs per server, 5 slices per index)
>>>> (searching and indexing at the same time, indexing once per week
>> during the weekend)
>>>> - each server has 4GB RAM, all servers are physical on seperate
>> machines
>>>> - RAM usage controlled by the processes
>>>> - total of 25.5 mio. docs (mainly metadata) from 1500 databases
>> worldwide
>>>> - index size is about 67GB per indexer --> about 402GB total
>>>> - about 3 qps at peek times
>>>> - with average search time of 0.05 seconds at peek times
>>>
>>> An average query time of 50 milliseconds isn't too bad. If the number
>> from your Solr setup below (39.5) is the QTime, then Solr thinks it is
>>> performing better, but Solr's QTime does not include absolutely
>> everything that hs to happen. Do you by chance have 95th and 99th
>> percentile
>>> query times for either system?
>>>
>>>> And here is now my current Solr setup:
>>>> - one master server (indexing only)
>>>> - two slave server (search only) but only one is online, the second
>> is fallback
>>>> - each server has 32GB RAM, all server are virtuell
>>>> (master on a seperate physical machine, both slaves together on a
>> physical machine)
>>>> - RAM usage is currently 20GB to java heap
>>>> - total of 31 mio. docs (all metadata) from 2000 databases worldwide
>>>> - index size is 156GB total
>>>> - search handler statistic report 0.6 average requests per second
>>>> - average time per request 39.5 (is that seconds?)
>>>> - building the index from scratch takes about 20 hours
>>>
>>> I can't tell whether you mean that each physical host has 32GB or
>> each VM has 32GB. You want to be sure that you are not oversubscribing
>> your
>>> memory. If you can get more memory in your machines, you really
>> should. Do you know whether that 0.6 seconds is most of the delay that
>> a user
>>> sees when making a search request, or are there other things going on
>> that contribute more delay? In our webapp, the Solr request time is
>>> usually small compared with everything else the server and the user's
>> browser are doing to render the results page. As much as I hate being
>> the
>>> tall pole in the tent, I look forward to the day when the developers
>> can change that balance.
>>>
>>>> The good thing is I have the ability to compare a commercial product
>> and
>>>> enterprise system to open source.
>>>>
>>>> I started with my simple Solr setup because of "kiss" (keep it
>> simple and stupid).
>>>> Actually it is doing excellent as single index on a single virtuell
>> server.
>>>> But the average time per request should be reduced now, thats why I
>> started
>>>> this discussion.
>>>> While searches with smaller Solr index size (3 mio. docs) showed
>> that it can
>>>> stand with FAST Search it now shows that its time to go with
>> sharding.
>>>> I think we are already far behind the point of search performance
>> crossover.
>>>>
>>>> What I hope to get with sharding:
>>>> - reduce time for building the index
>>>> - reduce average time per request
>>>
>>> You will probably achieve both of these things by sharding,
>> especially if you have a lot of CPU cores available. Like mine, your
>> query volume is
>>> very low, so the CPU cores are better utilized distributing the
>> search.
>>>
>>>> What I fear with sharding:
>>>> - i currently have master/slave, do I then have e.g. 3 master and 3
>> slaves?
>>>> - the query changes because of sharding (is there a search
>> distributor?)
>>>> - how to distribute the content the indexer with DIH on 3 server?
>>>> - anything else to think about while changing to sharding?
>>>
>>> I think sharding is probably a good idea for you, as long as you
>> don't lose redundancy. You can duplicate the FAST concept of a master
>> server,
>>> in a Solr core with no index. The solrconfig.xml for the core needs
>> to include the shards parameter. That core combined with those shards
>> will
>>> make up one complete index chain, and you need to have at least two
>> complete chains, running on separate physical hardware. A load balancer
>> will
>>> be critical. I use two small VMs on separate hosts with heartbeat and
>> haproxy for mine.
>>>
>>> Thanks,
>>> Shawn
>>>
RE: performance crossover between single index and sharding
Posted by Bob Sandiford <bo...@sirsidynix.com>.
Dumb question time - you are using a 64 bit Java, and not a 32 bit Java?
Bob Sandiford | Lead Software Engineer | SirsiDynix
P: 800.288.8020 X6943 | Bob.Sandiford@sirsidynix.com
www.sirsidynix.com
> -----Original Message-----
> From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
> Sent: Thursday, August 04, 2011 2:39 AM
> To: solr-user@lucene.apache.org
> Subject: Re: performance crossover between single index and sharding
>
> Hi Shawn,
>
> the 0.05 seconds for search time at peek times (3 qps) is my target for
> Solr.
> The numbers for solr are from Solr's statistic report page. So 39.5
> seconds
> average per request is definately to long and I have to change to
> sharding.
>
> For FAST system the numbers for the search dispatcher are:
> 0.042 sec elapsed per normal search, on avg.
> 0.053 sec average uncached normal search time (last 100 queries).
> 99.898% of searches using < 1 sec
> 99.999% of searches using < 3 sec
> 0.000% of all requests timed out
> 22454567.577 sec time up (that is 259 days)
>
> Is there a report page for those numbers for Solr?
>
> About the RAM, the 32GB RAM sind physical for each VM and the 20GB RAM
> are -Xmx for Java.
> Yesterday I noticed that we are running out of heap during replication
> so I have to
> increase -Xmx to about 22g.
>
> The reported 0.6 average requests per second seams to me right because
> the Solr system isn't under full load yet. The FAST system is still
> taking
> most of the load. I plan to switch completely to Solr after sharding is
> up and
> running stable. So there will be additional 3 qps to Solr at peek
> times.
>
> I don't know if a controlling master like FAST makes any sense for
> Solr.
> The small VMs with heartbeat and haproxy sounds great, must be on my
> todo list.
>
> But the biggest problem currently is, how to configure the DIH to split
> up the
> content to several indexer. Is there an indexing distributor?
>
> Regards,
> Bernd
>
>
> Am 03.08.2011 16:33, schrieb Shawn Heisey:
> > Replies inline.
> >
> > On 8/3/2011 2:24 AM, Bernd Fehling wrote:
> >> To show that I compare apples and oranges here are my previous FAST
> Search setup:
> >> - one master server (controlling, logging, search dispatcher)
> >> - six index server (4.25 mio docs per server, 5 slices per index)
> >> (searching and indexing at the same time, indexing once per week
> during the weekend)
> >> - each server has 4GB RAM, all servers are physical on seperate
> machines
> >> - RAM usage controlled by the processes
> >> - total of 25.5 mio. docs (mainly metadata) from 1500 databases
> worldwide
> >> - index size is about 67GB per indexer --> about 402GB total
> >> - about 3 qps at peek times
> >> - with average search time of 0.05 seconds at peek times
> >
> > An average query time of 50 milliseconds isn't too bad. If the number
> from your Solr setup below (39.5) is the QTime, then Solr thinks it is
> > performing better, but Solr's QTime does not include absolutely
> everything that hs to happen. Do you by chance have 95th and 99th
> percentile
> > query times for either system?
> >
> >> And here is now my current Solr setup:
> >> - one master server (indexing only)
> >> - two slave server (search only) but only one is online, the second
> is fallback
> >> - each server has 32GB RAM, all server are virtuell
> >> (master on a seperate physical machine, both slaves together on a
> physical machine)
> >> - RAM usage is currently 20GB to java heap
> >> - total of 31 mio. docs (all metadata) from 2000 databases worldwide
> >> - index size is 156GB total
> >> - search handler statistic report 0.6 average requests per second
> >> - average time per request 39.5 (is that seconds?)
> >> - building the index from scratch takes about 20 hours
> >
> > I can't tell whether you mean that each physical host has 32GB or
> each VM has 32GB. You want to be sure that you are not oversubscribing
> your
> > memory. If you can get more memory in your machines, you really
> should. Do you know whether that 0.6 seconds is most of the delay that
> a user
> > sees when making a search request, or are there other things going on
> that contribute more delay? In our webapp, the Solr request time is
> > usually small compared with everything else the server and the user's
> browser are doing to render the results page. As much as I hate being
> the
> > tall pole in the tent, I look forward to the day when the developers
> can change that balance.
> >
> >> The good thing is I have the ability to compare a commercial product
> and
> >> enterprise system to open source.
> >>
> >> I started with my simple Solr setup because of "kiss" (keep it
> simple and stupid).
> >> Actually it is doing excellent as single index on a single virtuell
> server.
> >> But the average time per request should be reduced now, thats why I
> started
> >> this discussion.
> >> While searches with smaller Solr index size (3 mio. docs) showed
> that it can
> >> stand with FAST Search it now shows that its time to go with
> sharding.
> >> I think we are already far behind the point of search performance
> crossover.
> >>
> >> What I hope to get with sharding:
> >> - reduce time for building the index
> >> - reduce average time per request
> >
> > You will probably achieve both of these things by sharding,
> especially if you have a lot of CPU cores available. Like mine, your
> query volume is
> > very low, so the CPU cores are better utilized distributing the
> search.
> >
> >> What I fear with sharding:
> >> - i currently have master/slave, do I then have e.g. 3 master and 3
> slaves?
> >> - the query changes because of sharding (is there a search
> distributor?)
> >> - how to distribute the content the indexer with DIH on 3 server?
> >> - anything else to think about while changing to sharding?
> >
> > I think sharding is probably a good idea for you, as long as you
> don't lose redundancy. You can duplicate the FAST concept of a master
> server,
> > in a Solr core with no index. The solrconfig.xml for the core needs
> to include the shards parameter. That core combined with those shards
> will
> > make up one complete index chain, and you need to have at least two
> complete chains, running on separate physical hardware. A load balancer
> will
> > be critical. I use two small VMs on separate hosts with heartbeat and
> haproxy for mine.
> >
> > Thanks,
> > Shawn
> >
Re: performance crossover between single index and sharding
Posted by Bernd Fehling <be...@uni-bielefeld.de>.
Hi Shawn,
the 0.05 seconds for search time at peek times (3 qps) is my target for Solr.
The numbers for solr are from Solr's statistic report page. So 39.5 seconds
average per request is definately to long and I have to change to sharding.
For FAST system the numbers for the search dispatcher are:
0.042 sec elapsed per normal search, on avg.
0.053 sec average uncached normal search time (last 100 queries).
99.898% of searches using < 1 sec
99.999% of searches using < 3 sec
0.000% of all requests timed out
22454567.577 sec time up (that is 259 days)
Is there a report page for those numbers for Solr?
About the RAM, the 32GB RAM sind physical for each VM and the 20GB RAM are -Xmx for Java.
Yesterday I noticed that we are running out of heap during replication so I have to
increase -Xmx to about 22g.
The reported 0.6 average requests per second seams to me right because
the Solr system isn't under full load yet. The FAST system is still taking
most of the load. I plan to switch completely to Solr after sharding is up and
running stable. So there will be additional 3 qps to Solr at peek times.
I don't know if a controlling master like FAST makes any sense for Solr.
The small VMs with heartbeat and haproxy sounds great, must be on my todo list.
But the biggest problem currently is, how to configure the DIH to split up the
content to several indexer. Is there an indexing distributor?
Regards,
Bernd
Am 03.08.2011 16:33, schrieb Shawn Heisey:
> Replies inline.
>
> On 8/3/2011 2:24 AM, Bernd Fehling wrote:
>> To show that I compare apples and oranges here are my previous FAST Search setup:
>> - one master server (controlling, logging, search dispatcher)
>> - six index server (4.25 mio docs per server, 5 slices per index)
>> (searching and indexing at the same time, indexing once per week during the weekend)
>> - each server has 4GB RAM, all servers are physical on seperate machines
>> - RAM usage controlled by the processes
>> - total of 25.5 mio. docs (mainly metadata) from 1500 databases worldwide
>> - index size is about 67GB per indexer --> about 402GB total
>> - about 3 qps at peek times
>> - with average search time of 0.05 seconds at peek times
>
> An average query time of 50 milliseconds isn't too bad. If the number from your Solr setup below (39.5) is the QTime, then Solr thinks it is
> performing better, but Solr's QTime does not include absolutely everything that hs to happen. Do you by chance have 95th and 99th percentile
> query times for either system?
>
>> And here is now my current Solr setup:
>> - one master server (indexing only)
>> - two slave server (search only) but only one is online, the second is fallback
>> - each server has 32GB RAM, all server are virtuell
>> (master on a seperate physical machine, both slaves together on a physical machine)
>> - RAM usage is currently 20GB to java heap
>> - total of 31 mio. docs (all metadata) from 2000 databases worldwide
>> - index size is 156GB total
>> - search handler statistic report 0.6 average requests per second
>> - average time per request 39.5 (is that seconds?)
>> - building the index from scratch takes about 20 hours
>
> I can't tell whether you mean that each physical host has 32GB or each VM has 32GB. You want to be sure that you are not oversubscribing your
> memory. If you can get more memory in your machines, you really should. Do you know whether that 0.6 seconds is most of the delay that a user
> sees when making a search request, or are there other things going on that contribute more delay? In our webapp, the Solr request time is
> usually small compared with everything else the server and the user's browser are doing to render the results page. As much as I hate being the
> tall pole in the tent, I look forward to the day when the developers can change that balance.
>
>> The good thing is I have the ability to compare a commercial product and
>> enterprise system to open source.
>>
>> I started with my simple Solr setup because of "kiss" (keep it simple and stupid).
>> Actually it is doing excellent as single index on a single virtuell server.
>> But the average time per request should be reduced now, thats why I started
>> this discussion.
>> While searches with smaller Solr index size (3 mio. docs) showed that it can
>> stand with FAST Search it now shows that its time to go with sharding.
>> I think we are already far behind the point of search performance crossover.
>>
>> What I hope to get with sharding:
>> - reduce time for building the index
>> - reduce average time per request
>
> You will probably achieve both of these things by sharding, especially if you have a lot of CPU cores available. Like mine, your query volume is
> very low, so the CPU cores are better utilized distributing the search.
>
>> What I fear with sharding:
>> - i currently have master/slave, do I then have e.g. 3 master and 3 slaves?
>> - the query changes because of sharding (is there a search distributor?)
>> - how to distribute the content the indexer with DIH on 3 server?
>> - anything else to think about while changing to sharding?
>
> I think sharding is probably a good idea for you, as long as you don't lose redundancy. You can duplicate the FAST concept of a master server,
> in a Solr core with no index. The solrconfig.xml for the core needs to include the shards parameter. That core combined with those shards will
> make up one complete index chain, and you need to have at least two complete chains, running on separate physical hardware. A load balancer will
> be critical. I use two small VMs on separate hosts with heartbeat and haproxy for mine.
>
> Thanks,
> Shawn
>
Re: performance crossover between single index and sharding
Posted by Shawn Heisey <so...@elyograg.org>.
Replies inline.
On 8/3/2011 2:24 AM, Bernd Fehling wrote:
> To show that I compare apples and oranges here are my previous FAST
> Search setup:
> - one master server (controlling, logging, search dispatcher)
> - six index server (4.25 mio docs per server, 5 slices per index)
> (searching and indexing at the same time, indexing once per week
> during the weekend)
> - each server has 4GB RAM, all servers are physical on seperate machines
> - RAM usage controlled by the processes
> - total of 25.5 mio. docs (mainly metadata) from 1500 databases worldwide
> - index size is about 67GB per indexer --> about 402GB total
> - about 3 qps at peek times
> - with average search time of 0.05 seconds at peek times
An average query time of 50 milliseconds isn't too bad. If the number
from your Solr setup below (39.5) is the QTime, then Solr thinks it is
performing better, but Solr's QTime does not include absolutely
everything that hs to happen. Do you by chance have 95th and 99th
percentile query times for either system?
> And here is now my current Solr setup:
> - one master server (indexing only)
> - two slave server (search only) but only one is online, the second is
> fallback
> - each server has 32GB RAM, all server are virtuell
> (master on a seperate physical machine, both slaves together on a
> physical machine)
> - RAM usage is currently 20GB to java heap
> - total of 31 mio. docs (all metadata) from 2000 databases worldwide
> - index size is 156GB total
> - search handler statistic report 0.6 average requests per second
> - average time per request 39.5 (is that seconds?)
> - building the index from scratch takes about 20 hours
I can't tell whether you mean that each physical host has 32GB or each
VM has 32GB. You want to be sure that you are not oversubscribing your
memory. If you can get more memory in your machines, you really
should. Do you know whether that 0.6 seconds is most of the delay that
a user sees when making a search request, or are there other things
going on that contribute more delay? In our webapp, the Solr request
time is usually small compared with everything else the server and the
user's browser are doing to render the results page. As much as I hate
being the tall pole in the tent, I look forward to the day when the
developers can change that balance.
> The good thing is I have the ability to compare a commercial product and
> enterprise system to open source.
>
> I started with my simple Solr setup because of "kiss" (keep it simple
> and stupid).
> Actually it is doing excellent as single index on a single virtuell
> server.
> But the average time per request should be reduced now, thats why I
> started
> this discussion.
> While searches with smaller Solr index size (3 mio. docs) showed that
> it can
> stand with FAST Search it now shows that its time to go with sharding.
> I think we are already far behind the point of search performance
> crossover.
>
> What I hope to get with sharding:
> - reduce time for building the index
> - reduce average time per request
You will probably achieve both of these things by sharding, especially
if you have a lot of CPU cores available. Like mine, your query volume
is very low, so the CPU cores are better utilized distributing the search.
> What I fear with sharding:
> - i currently have master/slave, do I then have e.g. 3 master and 3
> slaves?
> - the query changes because of sharding (is there a search distributor?)
> - how to distribute the content the indexer with DIH on 3 server?
> - anything else to think about while changing to sharding?
I think sharding is probably a good idea for you, as long as you don't
lose redundancy. You can duplicate the FAST concept of a master server,
in a Solr core with no index. The solrconfig.xml for the core needs to
include the shards parameter. That core combined with those shards will
make up one complete index chain, and you need to have at least two
complete chains, running on separate physical hardware. A load balancer
will be critical. I use two small VMs on separate hosts with heartbeat
and haproxy for mine.
Thanks,
Shawn
Re: performance crossover between single index and sharding
Posted by Bernd Fehling <be...@uni-bielefeld.de>.
On 02.08.2011 21:00, Shawn Heisey wrote:
> ...
> I did try some early tests with a single large index. Performance was pretty decent once it got warmed up, but I was worried about how it would
> perform under a heavy load, and how it would cope with frequent updates. I never really got very far with testing those fears, because the full
> rebuild time was unacceptable - at least 8 hours. The source database can keep up with six DIH instances reindexing at once, which completes
> much quicker than a single machine grabbing the entire database. I may increase the number of shards after I remove virtualization, but I'll
> need to fix a few limitations in my build system.
> ...
At first, thanks a lot to all answers and here is my setup.
I know that it is very difficult to give specific recommendations about this.
Because of changing from FAST Search to Solr I can state that Solr performs
very well, if not excellent.
To show that I compare apples and oranges here are my previous FAST Search setup:
- one master server (controlling, logging, search dispatcher)
- six index server (4.25 mio docs per server, 5 slices per index)
(searching and indexing at the same time, indexing once per week during the weekend)
- each server has 4GB RAM, all servers are physical on seperate machines
- RAM usage controlled by the processes
- total of 25.5 mio. docs (mainly metadata) from 1500 databases worldwide
- index size is about 67GB per indexer --> about 402GB total
- about 3 qps at peek times
- with average search time of 0.05 seconds at peek times
And here is now my current Solr setup:
- one master server (indexing only)
- two slave server (search only) but only one is online, the second is fallback
- each server has 32GB RAM, all server are virtuell
(master on a seperate physical machine, both slaves together on a physical machine)
- RAM usage is currently 20GB to java heap
- total of 31 mio. docs (all metadata) from 2000 databases worldwide
- index size is 156GB total
- search handler statistic report 0.6 average requests per second
- average time per request 39.5 (is that seconds?)
- building the index from scratch takes about 20 hours
The good thing is I have the ability to compare a commercial product and
enterprise system to open source.
I started with my simple Solr setup because of "kiss" (keep it simple and stupid).
Actually it is doing excellent as single index on a single virtuell server.
But the average time per request should be reduced now, thats why I started
this discussion.
While searches with smaller Solr index size (3 mio. docs) showed that it can
stand with FAST Search it now shows that its time to go with sharding.
I think we are already far behind the point of search performance crossover.
What I hope to get with sharding:
- reduce time for building the index
- reduce average time per request
What I fear with sharding:
- i currently have master/slave, do I then have e.g. 3 master and 3 slaves?
- the query changes because of sharding (is there a search distributor?)
- how to distribute the content the indexer with DIH on 3 server?
- anything else to think about while changing to sharding?
Conclusion:
- Solr can handle much more than 30 mio. docs of metadata in a single index
if java heap size is large enough. Have an eye on Lucenes fieldCache and
sorted fields, especially title (string) fields.
- The crossover in my case is somewhere between 3 mio. and 10 mio. docs
per index for Solr (compared to FAST Search). FAST recommends about 3 to 6 mio.
docs per 4GB RAM server for their system.
Anyone able to reduce my fears about sharding?
Thanks again for all your answers.
Regards
Bernd
--
*************************************************************
BASE - Bielefeld Academic Search Engine - www.base-search.net
*************************************************************
Re: performance crossover between single index and sharding
Posted by Shawn Heisey <so...@elyograg.org>.
On 8/2/2011 12:06 PM, Jonathan Rochkind wrote:
> What's the reasoning behind having three shards on one machine,
> instead of just combining those into one shard? Just curious. I had
> been thinking the point of shards was to get them on different
> machines, and there'd be no reason to have multiple shards on one
> machine.
I'd be interested in hearing Tom's answer as well, but my answer boils
down to the time it takes to do a full index rebuild and worries about
performance.
Because I'm in a virtualized environment, I effectively have three large
shards on each machine even though they are logically separate. When I
first got involved, we had a distributed EasyAsk index on 20 separate
low-end physical servers. That evolved into basically the same solution
with a smaller number of virtual machines, on a pair of very powerful
physical hosts. On this system, doing a full rebuild took nearly two
days and wasn't an atomic operation. The EasyAsk system (also based on
Lucene) was unable to deal with more than about 4 million documents per
machine (real or virtual). The only way to get acceptable performance
was distributed search. The cost of providing redundancy was too high,
so we didn't have any.
When we first started implementing Solr, we assumed from our previous
experience that we'd need distributed search, especially if query volume
were to go up. For that reason, we continued our virtualization model,
but with only seven shards - six large "static" shards and a smaller
"incremental" shard to hold data less than a week old. This is where we
are now, and performance is MUCH better than the old solution. The low
shard count made redundancy affordable, so we now have that too.
At the time Solr was first implemented, we could rebuild the entire
index in about two hours and swap it into place all at once. Our index
has grown enough since then that it takes a little less than three
hours, which is still pretty quick for 60 million documents.
I did try some early tests with a single large index. Performance was
pretty decent once it got warmed up, but I was worried about how it
would perform under a heavy load, and how it would cope with frequent
updates. I never really got very far with testing those fears, because
the full rebuild time was unacceptable - at least 8 hours. The source
database can keep up with six DIH instances reindexing at once, which
completes much quicker than a single machine grabbing the entire
database. I may increase the number of shards after I remove
virtualization, but I'll need to fix a few limitations in my build system.
Thanks,
Shawn
Re: performance crossover between single index and sharding
Posted by Ken Krugler <kk...@transpac.com>.
With low qps and multi-core servers, I believe one reason to have multiple shards on one server is to provide better parallelism for a request, and thus reduce your response time.
-- Ken
On Aug 2, 2011, at 11:06am, Jonathan Rochkind wrote:
> What's the reasoning behind having three shards on one machine, instead of just combining those into one shard? Just curious. I had been thinking the point of shards was to get them on different machines, and there'd be no reason to have multiple shards on one machine.
>
> On 8/2/2011 1:59 PM, Burton-West, Tom wrote:
>> Hi Markus,
>>
>> Just as a data point for a very large sharded index, we have the full text of 9.3 million books with an index size of about 6+ TB spread over 12 shards on 4 machines. Each machine has 3 shards. The size of each shard ranges between 475GB and 550GB. We are definitely I/O bound. Our machines have 144GB of memory with about 16GB dedicated to the tomcat instance running the 3 Solr instances, which leaves about 120 GB (or 40GB per shard) for the OS disk cache. We release a new index every morning and then warm the caches with several thousand queries. I probably should add that our disk storage is a very high performance Isilon appliance that has over 500 drives and every block of every file is striped over no less than 14 different drives. (See blog for details *)
>>
>> We have a very low number of queries per second (0.3-2 qps) and our modest response time goal is to keep 99th percentile response time for our application (i.e. Solr + application) under 10 seconds.
>>
>> Our current performance statistics are:
>>
>> average response time 300 ms
>> median response time 113 ms
>> 90th percentile 663 ms
>> 95th percentile 1,691 ms
>>
>> We had plans to do some performance testing to determine the optimum shard size and optimum number of shards per machine, but that has remained on the back burner for a long time as other higher priority items keep pushing it down on the todo list.
>>
>> We would be really interested to hear about the experiences of people who have so many shards that the overhead of distributing the queries, and consolidating/merging the responses becomes a serious issue.
>>
>>
>> Tom Burton-West
>>
>> http://www.hathitrust.org/blogs/large-scale-search
>>
>> * http://www.hathitrust.org/blogs/large-scale-search/scaling-large-scale-search-500000-volumes-5-million-volumes-and-beyond
>>
>> -----Original Message-----
>> From: Markus Jelsma [mailto:markus.jelsma@openindex.io]
>> Sent: Tuesday, August 02, 2011 12:33 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: performance crossover between single index and sharding
>>
>> Actually, i do worry about it. Would be marvelous if someone could provide
>> some metrics for an index of many terabytes.
>>
>>> [..] At some extreme point there will be diminishing
>>> returns and a performance decrease, but I wouldn't worry about that at all
>>> until you've got many terabytes -- I don't know how many but don't worry
>>> about it.
>>>
>>> ~ David
>>>
>>> -----
>>> Author: https://www.packtpub.com/solr-1-4-enterprise-search-server/book
>>> --
>>> View this message in context:
>>> http://lucene.472066.n3.nabble.com/performance-crossover-between-single-in
>>> dex-and-sharding-tp3218561p3219397.html Sent from the Solr - User mailing
>>> list archive at Nabble.com.
--------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
custom data mining solutions
Re: performance crossover between single index and sharding
Posted by Jonathan Rochkind <ro...@jhu.edu>.
What's the reasoning behind having three shards on one machine, instead
of just combining those into one shard? Just curious. I had been
thinking the point of shards was to get them on different machines, and
there'd be no reason to have multiple shards on one machine.
On 8/2/2011 1:59 PM, Burton-West, Tom wrote:
> Hi Markus,
>
> Just as a data point for a very large sharded index, we have the full text of 9.3 million books with an index size of about 6+ TB spread over 12 shards on 4 machines. Each machine has 3 shards. The size of each shard ranges between 475GB and 550GB. We are definitely I/O bound. Our machines have 144GB of memory with about 16GB dedicated to the tomcat instance running the 3 Solr instances, which leaves about 120 GB (or 40GB per shard) for the OS disk cache. We release a new index every morning and then warm the caches with several thousand queries. I probably should add that our disk storage is a very high performance Isilon appliance that has over 500 drives and every block of every file is striped over no less than 14 different drives. (See blog for details *)
>
> We have a very low number of queries per second (0.3-2 qps) and our modest response time goal is to keep 99th percentile response time for our application (i.e. Solr + application) under 10 seconds.
>
> Our current performance statistics are:
>
> average response time 300 ms
> median response time 113 ms
> 90th percentile 663 ms
> 95th percentile 1,691 ms
>
> We had plans to do some performance testing to determine the optimum shard size and optimum number of shards per machine, but that has remained on the back burner for a long time as other higher priority items keep pushing it down on the todo list.
>
> We would be really interested to hear about the experiences of people who have so many shards that the overhead of distributing the queries, and consolidating/merging the responses becomes a serious issue.
>
>
> Tom Burton-West
>
> http://www.hathitrust.org/blogs/large-scale-search
>
> * http://www.hathitrust.org/blogs/large-scale-search/scaling-large-scale-search-500000-volumes-5-million-volumes-and-beyond
>
> -----Original Message-----
> From: Markus Jelsma [mailto:markus.jelsma@openindex.io]
> Sent: Tuesday, August 02, 2011 12:33 PM
> To: solr-user@lucene.apache.org
> Subject: Re: performance crossover between single index and sharding
>
> Actually, i do worry about it. Would be marvelous if someone could provide
> some metrics for an index of many terabytes.
>
>> [..] At some extreme point there will be diminishing
>> returns and a performance decrease, but I wouldn't worry about that at all
>> until you've got many terabytes -- I don't know how many but don't worry
>> about it.
>>
>> ~ David
>>
>> -----
>> Author: https://www.packtpub.com/solr-1-4-enterprise-search-server/book
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/performance-crossover-between-single-in
>> dex-and-sharding-tp3218561p3219397.html Sent from the Solr - User mailing
>> list archive at Nabble.com.
RE: performance crossover between single index and sharding
Posted by "Burton-West, Tom" <tb...@umich.edu>.
Hi Markus,
Just as a data point for a very large sharded index, we have the full text of 9.3 million books with an index size of about 6+ TB spread over 12 shards on 4 machines. Each machine has 3 shards. The size of each shard ranges between 475GB and 550GB. We are definitely I/O bound. Our machines have 144GB of memory with about 16GB dedicated to the tomcat instance running the 3 Solr instances, which leaves about 120 GB (or 40GB per shard) for the OS disk cache. We release a new index every morning and then warm the caches with several thousand queries. I probably should add that our disk storage is a very high performance Isilon appliance that has over 500 drives and every block of every file is striped over no less than 14 different drives. (See blog for details *)
We have a very low number of queries per second (0.3-2 qps) and our modest response time goal is to keep 99th percentile response time for our application (i.e. Solr + application) under 10 seconds.
Our current performance statistics are:
average response time 300 ms
median response time 113 ms
90th percentile 663 ms
95th percentile 1,691 ms
We had plans to do some performance testing to determine the optimum shard size and optimum number of shards per machine, but that has remained on the back burner for a long time as other higher priority items keep pushing it down on the todo list.
We would be really interested to hear about the experiences of people who have so many shards that the overhead of distributing the queries, and consolidating/merging the responses becomes a serious issue.
Tom Burton-West
http://www.hathitrust.org/blogs/large-scale-search
* http://www.hathitrust.org/blogs/large-scale-search/scaling-large-scale-search-500000-volumes-5-million-volumes-and-beyond
-----Original Message-----
From: Markus Jelsma [mailto:markus.jelsma@openindex.io]
Sent: Tuesday, August 02, 2011 12:33 PM
To: solr-user@lucene.apache.org
Subject: Re: performance crossover between single index and sharding
Actually, i do worry about it. Would be marvelous if someone could provide
some metrics for an index of many terabytes.
> [..] At some extreme point there will be diminishing
> returns and a performance decrease, but I wouldn't worry about that at all
> until you've got many terabytes -- I don't know how many but don't worry
> about it.
>
> ~ David
>
> -----
> Author: https://www.packtpub.com/solr-1-4-enterprise-search-server/book
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/performance-crossover-between-single-in
> dex-and-sharding-tp3218561p3219397.html Sent from the Solr - User mailing
> list archive at Nabble.com.
Re: performance crossover between single index and sharding
Posted by Markus Jelsma <ma...@openindex.io>.
Actually, i do worry about it. Would be marvelous if someone could provide
some metrics for an index of many terabytes.
> [..] At some extreme point there will be diminishing
> returns and a performance decrease, but I wouldn't worry about that at all
> until you've got many terabytes -- I don't know how many but don't worry
> about it.
>
> ~ David
>
> -----
> Author: https://www.packtpub.com/solr-1-4-enterprise-search-server/book
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/performance-crossover-between-single-in
> dex-and-sharding-tp3218561p3219397.html Sent from the Solr - User mailing
> list archive at Nabble.com.
Re: performance crossover between single index and sharding
Posted by "David Smiley (@MITRE.org)" <DS...@mitre.org>.
That's a fantastic answer, Shawn.
To more directly answer Bernd's question: Bernard, shard your data once
you've done reasonable performance optimizations to your single core index
setup (see Chapter 9 of my book) and the query response time isn't meeting
your requirements in spite of this. Solr scales pretty darned well
horizontally -- so as you shard your data more and more, the query
responses will get faster. At some extreme point there will be diminishing
returns and a performance decrease, but I wouldn't worry about that at all
until you've got many terabytes -- I don't know how many but don't worry
about it.
~ David
-----
Author: https://www.packtpub.com/solr-1-4-enterprise-search-server/book
--
View this message in context: http://lucene.472066.n3.nabble.com/performance-crossover-between-single-index-and-sharding-tp3218561p3219397.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: performance crossover between single index and sharding
Posted by Shawn Heisey <so...@elyograg.org>.
On 8/2/2011 4:44 AM, Bernd Fehling wrote:
> Is there any knowledge on this list about the performance
> crossover between a single index and sharding and
> when to change from a single index to sharding?
>
> E.g. if index size is larger than 150GB and num of docs is
> more than 25 mio. then it is better to change from single index
> to sharding and have two shards.
> Or something like this...
>
> Sure, solr might even handle 50 mio. docs but performance is going down
> and a sharded system with distributed search will be faster than
> a single index, or not?
The answer I've always seen here boils down to "it depends on a large
number of variables unique to every situation." The nature of your data
will affect things, like the number of fields, number of unique terms
per field, etc. If you have really complicated queries, that will slow
things down.
Probably the greatest limiting factor is memory. Having enough free
memory to fit the entire index into the operating system's disk cache is
the best thing you can do for performance. This is memory over and
above whatever you give to your Java heap. If you have a 150GB index
and you can afford machines with at least 192GB of RAM, a single index
would perform very well, once it is warmed up. Performance on a cold
index would not be very good. In a sharded scenario, you want to try
and size each machine so that its piece fits into RAM.
Next would be disk I/O. Any data that won't fit in the disk cache must
be retrieved from disk, which is typically the weakest link in the
chain. If you can put your index on solid state disks, that's almost as
good as having the index entirely in memory. Performance on a cold
index with SSD would be incredible.
Having a lot of high speed CPU available will help, but not as much as
memory and I/O.
Index rebuild time is another consideration that might lead you to go
distributed, as long as your data source can keep up with multiple readers.
My own index is too big to fit in RAM, even sharded. Each of the six
large shards is getting close to 19GB. Each machine has 14GB of RAM
(it's a virtual environment with three large shards per physical host)
and has 3GB allocated to Java. I am in the process of upgrading the
memory, at which point it will fit, but our growth will exceed the
maximum server memory again in the next year or so. I have plans to
eliminate the virtualization and have three shards in cores on each server.
I know this isn't really what you were looking for, but there are no
simple answers to your question.
Thanks,
Shawn