You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Andrew Nagy <an...@villanova.edu> on 2007/08/30 18:51:05 UTC
performance questions
Here are a few SOLR performance questions:
1. I have noticed with 500,000+ records that my facets run quite fast regarding my dataset when there is a large number of matches, but on a small result set (say 10 - 50) the facet queries become very slow. Any suggestions as to how to improve this?
2. Someone asked me if SOLR utilizes anything like a "stored procedure" to make queries faster. Does SOLR support anything such as this?
Thanks
Andrew
RE: performance questions
Posted by Jonathan Woods <jo...@scintillance.com>.
Only if you think the rest of Solr would be better written in JRuby too!
> -----Original Message-----
> From: Erik Hatcher [mailto:erik@ehatchersolutions.com]
> Sent: 31 August 2007 02:57
> To: solr-user@lucene.apache.org
> Subject: Re: performance questions
>
>
> On Aug 30, 2007, at 6:31 PM, Mike Klaas wrote:
> > Another reason why people use stored procs is to prevent multiple
> > round-trips in a multi-stage query operation. This is exactly what
> > complex RequestHandlers do (and the equivalent to a custom
> stored proc
> > would be writing your own handler).
>
> And we should be writing those handlers in JRuby ;) Who's with me?
>
> Erik
>
>
>
>
Re: performance questions
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Aug 30, 2007, at 11:37 PM, Walter Underwood wrote:
> Sorry dude, I'm pining for Python and coding in Java. --wunder
No need to be sorry.... well, unless you do really enjoy forced
whitespace and __init__ ugliness. :)
Why don't we tie in scripting engine support into Solr via BSF or the
javax extension methods?
What about the new release of jython?
Erik
>
> On 8/30/07 6:57 PM, "Erik Hatcher" <er...@ehatchersolutions.com> wrote:
>
>>
>> On Aug 30, 2007, at 6:31 PM, Mike Klaas wrote:
>>> Another reason why people use stored procs is to prevent multiple
>>> round-trips in a multi-stage query operation. This is exactly what
>>> complex RequestHandlers do (and the equivalent to a custom stored
>>> proc would be writing your own handler).
>>
>> And we should be writing those handlers in JRuby ;) Who's with me?
>>
>> Erik
>>
Re: performance questions
Posted by Walter Underwood <wu...@netflix.com>.
Sorry dude, I'm pining for Python and coding in Java. --wunder
On 8/30/07 6:57 PM, "Erik Hatcher" <er...@ehatchersolutions.com> wrote:
>
> On Aug 30, 2007, at 6:31 PM, Mike Klaas wrote:
>> Another reason why people use stored procs is to prevent multiple
>> round-trips in a multi-stage query operation. This is exactly what
>> complex RequestHandlers do (and the equivalent to a custom stored
>> proc would be writing your own handler).
>
> And we should be writing those handlers in JRuby ;) Who's with me?
>
> Erik
>
Re: performance questions
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Aug 30, 2007, at 6:31 PM, Mike Klaas wrote:
> Another reason why people use stored procs is to prevent multiple
> round-trips in a multi-stage query operation. This is exactly what
> complex RequestHandlers do (and the equivalent to a custom stored
> proc would be writing your own handler).
And we should be writing those handlers in JRuby ;) Who's with me?
Erik
Re: performance questions
Posted by Mike Klaas <mi...@gmail.com>.
On 30-Aug-07, at 3:18 PM, Chris Hostetter wrote:
>
>> 2. Someone asked me if SOLR utilizes anything like a "stored
>> procedure" to make queries faster. Does SOLR support anything
>> such as this?
>
> it's kind of an apples vs orange-juice comparison, ut typcailly
> when people talk about DB stored procedures being faster then raw
> SQL they are refering tohte fact that the SQL in the stored proc is
> compiled only once, and a query plan is produced which is then
> reused each time the proc is called with various inputs.
>
> in that regard no, Solr does not have any equivilent functinality.
> the query parser that "compiles" string input into Query objects to
> execute does not save existing Query structures for reuse with
> different input values. the Query=DocSet mappings from previous
> queries are cached and reused however ... so two people searching
> for hte exact same thing without any index modifications between
> them will result in the second person getting very fast
> performance ... i don't know of any DB that has anything like
> that. Solr's ability to "autowarm" recent queies when doing a
> commit is also something i've never heard of any type of DB doing.
Modern SQL DBs have (typically small) query caches that function
similarly to Solr's query/document cache, and row-level caches that
function similarly to the OS disk cache in Solr. Though I don't know
if there is anything quite like filter caches in most DBs.
Another reason why people use stored procs is to prevent multiple
round-trips in a multi-stage query operation. This is exactly what
complex RequestHandlers do (and the equivalent to a custom stored
proc would be writing your own handler).
-Mike
Re: performance questions
Posted by Chris Hostetter <ho...@fucit.org>.
> 2. Someone asked me if SOLR utilizes anything like a "stored procedure"
> to make queries faster. Does SOLR support anything such as this?
it's kind of an apples vs orange-juice comparison, ut typcailly when
people talk about DB stored procedures being faster then raw SQL they are
refering tohte fact that the SQL in the stored proc is compiled only once,
and a query plan is produced which is then reused each time the proc is
called with various inputs.
in that regard no, Solr does not have any equivilent functinality. the
query parser that "compiles" string input into Query objects to execute
does not save existing Query structures for reuse with different input
values. the Query=DocSet mappings from previous queries are cached and
reused however ... so two people searching for hte exact same thing
without any index modifications between them will result in the second
person getting very fast performance ... i don't know of any DB that has
anything like that. Solr's ability to "autowarm" recent queies when doing
a commit is also something i've never heard of any type of DB doing.
...both of which just illustrate that while DBs focus on being database,
Solr focuses on being a search service. :)
-Hoss
Re: performance questions
Posted by Mike Klaas <mi...@gmail.com>.
On 30-Aug-07, at 9:51 AM, Andrew Nagy wrote:
> Here are a few SOLR performance questions:
>
> 1. I have noticed with 500,000+ records that my facets run quite
> fast regarding my dataset when there is a large number of matches,
> but on a small result set (say 10 - 50) the facet queries become
> very slow. Any suggestions as to how to improve this?
Maybe a problem with HashSets? Try reducing this value to zero:
<!-- This entry enables an int hash representation for filters (DocSets)
when the number of items in the set is less than maxSize.
For smaller
sets, this representation is more memory efficient, more
efficient to
iterate over, and faster to take intersections. -->
<HashDocSet maxSize="3000" loadFactor="0.75"/>
-Mike