You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Britske <gb...@gmail.com> on 2007/12/27 15:45:45 UTC
big perf-difference between solr-server vs. SOlrJ
req.process(solrserver)
Hi,
I am using SolrJ to communicate with SOLR. My Solr-queries perform within
range (between 50 ms and 300 ms) by looking at the solr log as ouputted on
my (windows) commandline.
However I discovered that the following command at all times takes
significantly longer than the number outputted in the solr-log, (sometimes
about 400% longer):
SorrQuery query = /* some query */
QueryRequest req = new QueryRequest(query);
req.process(solrServer); //solrServer is an instance of
CommonsHttpSolrServer
Of course, this includes the time to parse and transfer the response (about
10-12k) from the solr-server to solrJ, but I can't imagine that it can all
be contributed to this.
I would really appreciate anyone giving me some pointers as to what would
make up this difference. And/Or whether this difference seems normal or not.
As a refeference ,some figures I've seen are:
30 ms on server 125ms on solrJ
250ms on server 550ms on solrJ.
at all times the response is between 10-12k which includes facetfields and
facetqueries. (A facetfield takes has up to a max of about 100 values).
Thanks in advance,
Geert-Jan
--
View this message in context: http://www.nabble.com/big-perf-difference-between-solr-server-vs.--SOlrJ-req.process%28solrserver%29-tp14513964p14513964.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: big perf-difference between solr-server vs. SOlrJ req.process(solrserver)
Posted by "Norskog, Lance" <la...@divvio.com>.
As a reference:
I have several million records where are about 20 fields. One of them is
100-1k bytes, and the rest are 20-50 bytes. There is a reliable 5%
performance difference between pulling just the unique key field and
pulling all of the fields.
-----Original Message-----
From: Geert-Jan Brits [mailto:gbrits@gmail.com]
Sent: Thursday, December 27, 2007 8:44 AM
To: solr-user@lucene.apache.org
Subject: Re: big perf-difference between solr-server vs. SOlrJ
req.process(solrserver)
yeah, that makes sense.
so, in in all, could scanning all the fields and loading the 10 fields
add up to cost about the same or even more as performing the intial
query? (Just making sure)
I am wondering if the following change to the schema would help in this
case:
current setup:
It's possible to have up to 2000 product-variants.
each product-variant has:
- 1 price field (stored / indexed)
- 1 multivalued field which contains product-variant characteristics
(strored / not indexed).
This adds up to the 4000 fields described. Moreover there are some
fields on the product level but these would contibute just a tiny bit to
the overall scanning / loading costs (about 50 -stored and indexed-
fields in total)
possible new setup (only the changes) :
- index but not store the price-field.
- store the price as just another one of the product-variant
characteristics in the multivalued product-variant field.
as a result this would bring back the maximum number of stored fields to
about 2050 from 4050 and thereby about halving scanning / loading costs
while leaving the current quering-costs intact.
Indexing costs would increase a bit.
Would you expect the same performance gain?
Thanks,
Geert-Jan
2007/12/27, Yonik Seeley <yo...@apache.org>:
>
> On Dec 27, 2007 11:01 AM, Britske <gb...@gmail.com> wrote:
> > after inspecting solrconfig.xml I see that I already have enabled
> > lazy
> field
> > loading by:
> > <enableLazyFieldLoading>true</enableLazyFieldLoading> (I guess it
> > was enabled by default)
> >
> > Since any query returns about 10 fields (which differ from query to
> query) ,
> > would this mean that only these 10 of about 2000-4000 fields are
> retrieved /
> > loaded?
>
> Yes, but that's not the whole story.
> Lucene stores all of the fields back-to-back with no index (there is
> no random access to particular stored fields)... so all of the fields
> must be at least scanned.
>
> -Yonik
>
Re: big perf-difference between solr-server vs. SOlrJ req.process(solrserver)
Posted by Geert-Jan Brits <gb...@gmail.com>.
yeah, that makes sense.
so, in in all, could scanning all the fields and loading the 10 fields add
up to cost about the same or even more as performing the intial query? (Just
making sure)
I am wondering if the following change to the schema would help in this
case:
current setup:
It's possible to have up to 2000 product-variants.
each product-variant has:
- 1 price field (stored / indexed)
- 1 multivalued field which contains product-variant characteristics
(strored / not indexed).
This adds up to the 4000 fields described. Moreover there are some fields on
the product level but these would contibute just a tiny bit to the overall
scanning / loading costs (about 50 -stored and indexed- fields in total)
possible new setup (only the changes) :
- index but not store the price-field.
- store the price as just another one of the product-variant characteristics
in the multivalued product-variant field.
as a result this would bring back the maximum number of stored fields to
about 2050 from 4050 and thereby about halving scanning / loading costs
while leaving the current quering-costs intact.
Indexing costs would increase a bit.
Would you expect the same performance gain?
Thanks,
Geert-Jan
2007/12/27, Yonik Seeley <yo...@apache.org>:
>
> On Dec 27, 2007 11:01 AM, Britske <gb...@gmail.com> wrote:
> > after inspecting solrconfig.xml I see that I already have enabled lazy
> field
> > loading by:
> > <enableLazyFieldLoading>true</enableLazyFieldLoading> (I guess it was
> > enabled by default)
> >
> > Since any query returns about 10 fields (which differ from query to
> query) ,
> > would this mean that only these 10 of about 2000-4000 fields are
> retrieved /
> > loaded?
>
> Yes, but that's not the whole story.
> Lucene stores all of the fields back-to-back with no index (there is
> no random access to particular stored fields)... so all of the fields
> must be at least scanned.
>
> -Yonik
>
Re: big perf-difference between solr-server vs. SOlrJ req.process(solrserver)
Posted by Yonik Seeley <yo...@apache.org>.
On Dec 31, 2007 8:58 AM, Britske <gb...@gmail.com> wrote:
> Moreover, I realized that I'm using an xsl-transform in the post-processing
> phase. This would contribute to the high cost I'm seeing as well I think.
> Can this XSL-transform in general be considered small in relation to the
> abovementioned costs?
The XSLT transform could possibly be expensive. The easiest thing is
to try it without the transform and see.
-Yonik
Re: big perf-difference between solr-server vs. SOlrJ
req.process(solrserver)
Posted by Britske <gb...@gmail.com>.
I imagine then that this "scanning-cost" is proportional to the number of
stored fields, correct?
I tested this with generating a second index with 1/10th of the
product-variants (and thus 1/10th) of the stored fields. However I really
don't see the expected (at least by me) drop in post-processing time (which
includes lazy loading the needed fields and scanning all the stored fields).
Moreover, I realized that I'm using an xsl-transform in the post-processing
phase. This would contribute to the high cost I'm seeing as well I think.
Can this XSL-transform in general be considered small in relation to the
abovementioned costs?
Thanks,
Geert-Jan
Yonik Seeley wrote:
>
> On Dec 27, 2007 11:01 AM, Britske <gb...@gmail.com> wrote:
>> after inspecting solrconfig.xml I see that I already have enabled lazy
>> field
>> loading by:
>> <enableLazyFieldLoading>true</enableLazyFieldLoading> (I guess it was
>> enabled by default)
>>
>> Since any query returns about 10 fields (which differ from query to
>> query) ,
>> would this mean that only these 10 of about 2000-4000 fields are
>> retrieved /
>> loaded?
>
> Yes, but that's not the whole story.
> Lucene stores all of the fields back-to-back with no index (there is
> no random access to particular stored fields)... so all of the fields
> must be at least scanned.
>
> -Yonik
>
>
--
View this message in context: http://www.nabble.com/big-perf-difference-between-solr-server-vs.--SOlrJ-req.process%28solrserver%29-tp14513964p14557779.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: big perf-difference between solr-server vs. SOlrJ req.process(solrserver)
Posted by Yonik Seeley <yo...@apache.org>.
On Dec 27, 2007 11:01 AM, Britske <gb...@gmail.com> wrote:
> after inspecting solrconfig.xml I see that I already have enabled lazy field
> loading by:
> <enableLazyFieldLoading>true</enableLazyFieldLoading> (I guess it was
> enabled by default)
>
> Since any query returns about 10 fields (which differ from query to query) ,
> would this mean that only these 10 of about 2000-4000 fields are retrieved /
> loaded?
Yes, but that's not the whole story.
Lucene stores all of the fields back-to-back with no index (there is
no random access to particular stored fields)... so all of the fields
must be at least scanned.
-Yonik
Re: big perf-difference between solr-server vs. SOlrJ
req.process(solrserver)
Posted by Britske <gb...@gmail.com>.
after inspecting solrconfig.xml I see that I already have enabled lazy field
loading by:
<enableLazyFieldLoading>true</enableLazyFieldLoading> (I guess it was
enabled by default)
Since any query returns about 10 fields (which differ from query to query) ,
would this mean that only these 10 of about 2000-4000 fields are retrieved /
loaded?
Thanks,
Geert-Jan
Erick Erickson wrote:
>
> From a Lucene perspective, it's certainly possible to do lazy field
> loading. That is, when loading a document you can determine at
> run time what fields to load, even on a per-document basis. I'm
> not entirely sure how to accomplish this in Solr, but I'd give
> long odds that there's a way.....
>
> I did a writeup on this on the Wiki, see:
>
> http://wiki.apache.org/lucene-java/FieldSelectorPerformance?highlight=%28fieldselectorperformance%29
>
>
> The title is FieldSelectorPerformance if you need to search the Wiki...
>
> Best
> Erick
>
> On Dec 27, 2007 10:28 AM, Britske <gb...@gmail.com> wrote:
>
>>
>>
>>
>> Yonik Seeley wrote:
>> >
>> > On Dec 27, 2007 9:45 AM, Britske <gb...@gmail.com> wrote:
>> >> I am using SolrJ to communicate with SOLR. My Solr-queries perform
>> within
>> >> range (between 50 ms and 300 ms) by looking at the solr log as
>> ouputted
>> >> on
>> >> my (windows) commandline.
>> >>
>> >> However I discovered that the following command at all times takes
>> >> significantly longer than the number outputted in the solr-log,
>> >> (sometimes
>> >> about 400% longer):
>> >
>> > It's probably due to stored field retrieval.
>> > The time in the response includes everything except the time to write
>> > the response (since it appears at the beginning). Writing the
>> > response involves reading the stored fields of documents (this was
>> > done to allow one to stream a large number of documents w/o having
>> > them all in memory).
>> >
>> > SolrJ's parsing of the response should be a relatively small constant
>> > cost.
>> >
>> > -Yonik
>> >
>> >
>>
>> Is it normal to see this much time taken in stored field retrieval? And
>> where would I start to make sure that it is indeed caused by stored field
>> retrieval?
>>
>> It seems quite much to me, although I have kind if an out of the ordinary
>> setup with between 2000-4000 stored fields per document. By far the
>> largest
>> part is taken by various 'product-variants' and their respective prices
>> (indexed field) and other characteristics (stored only).
>> However only about 10 stored fields per document are returned for any
>> possible query.
>>
>> Would the time taken still include iterating the non-returned fields (of
>> which there are many in my case), or are only the returned fields
>> retrieved
>> in a map-like implementation?
>>
>> Thanks,
>> Geert-Jan
>> --
>> View this message in context:
>> http://www.nabble.com/big-perf-difference-between-solr-server-vs.--SOlrJ-req.process%28solrserver%29-tp14513964p14514441.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://www.nabble.com/big-perf-difference-between-solr-server-vs.--SOlrJ-req.process%28solrserver%29-tp14513964p14514852.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: big perf-difference between solr-server vs. SOlrJ req.process(solrserver)
Posted by Erick Erickson <er...@gmail.com>.
>From a Lucene perspective, it's certainly possible to do lazy field
loading. That is, when loading a document you can determine at
run time what fields to load, even on a per-document basis. I'm
not entirely sure how to accomplish this in Solr, but I'd give
long odds that there's a way.....
I did a writeup on this on the Wiki, see:
http://wiki.apache.org/lucene-java/FieldSelectorPerformance?highlight=%28fieldselectorperformance%29
The title is FieldSelectorPerformance if you need to search the Wiki...
Best
Erick
On Dec 27, 2007 10:28 AM, Britske <gb...@gmail.com> wrote:
>
>
>
> Yonik Seeley wrote:
> >
> > On Dec 27, 2007 9:45 AM, Britske <gb...@gmail.com> wrote:
> >> I am using SolrJ to communicate with SOLR. My Solr-queries perform
> within
> >> range (between 50 ms and 300 ms) by looking at the solr log as ouputted
> >> on
> >> my (windows) commandline.
> >>
> >> However I discovered that the following command at all times takes
> >> significantly longer than the number outputted in the solr-log,
> >> (sometimes
> >> about 400% longer):
> >
> > It's probably due to stored field retrieval.
> > The time in the response includes everything except the time to write
> > the response (since it appears at the beginning). Writing the
> > response involves reading the stored fields of documents (this was
> > done to allow one to stream a large number of documents w/o having
> > them all in memory).
> >
> > SolrJ's parsing of the response should be a relatively small constant
> > cost.
> >
> > -Yonik
> >
> >
>
> Is it normal to see this much time taken in stored field retrieval? And
> where would I start to make sure that it is indeed caused by stored field
> retrieval?
>
> It seems quite much to me, although I have kind if an out of the ordinary
> setup with between 2000-4000 stored fields per document. By far the
> largest
> part is taken by various 'product-variants' and their respective prices
> (indexed field) and other characteristics (stored only).
> However only about 10 stored fields per document are returned for any
> possible query.
>
> Would the time taken still include iterating the non-returned fields (of
> which there are many in my case), or are only the returned fields
> retrieved
> in a map-like implementation?
>
> Thanks,
> Geert-Jan
> --
> View this message in context:
> http://www.nabble.com/big-perf-difference-between-solr-server-vs.--SOlrJ-req.process%28solrserver%29-tp14513964p14514441.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
Re: big perf-difference between solr-server vs. SOlrJ
req.process(solrserver)
Posted by Britske <gb...@gmail.com>.
Yonik Seeley wrote:
>
> On Dec 27, 2007 9:45 AM, Britske <gb...@gmail.com> wrote:
>> I am using SolrJ to communicate with SOLR. My Solr-queries perform within
>> range (between 50 ms and 300 ms) by looking at the solr log as ouputted
>> on
>> my (windows) commandline.
>>
>> However I discovered that the following command at all times takes
>> significantly longer than the number outputted in the solr-log,
>> (sometimes
>> about 400% longer):
>
> It's probably due to stored field retrieval.
> The time in the response includes everything except the time to write
> the response (since it appears at the beginning). Writing the
> response involves reading the stored fields of documents (this was
> done to allow one to stream a large number of documents w/o having
> them all in memory).
>
> SolrJ's parsing of the response should be a relatively small constant
> cost.
>
> -Yonik
>
>
Is it normal to see this much time taken in stored field retrieval? And
where would I start to make sure that it is indeed caused by stored field
retrieval?
It seems quite much to me, although I have kind if an out of the ordinary
setup with between 2000-4000 stored fields per document. By far the largest
part is taken by various 'product-variants' and their respective prices
(indexed field) and other characteristics (stored only).
However only about 10 stored fields per document are returned for any
possible query.
Would the time taken still include iterating the non-returned fields (of
which there are many in my case), or are only the returned fields retrieved
in a map-like implementation?
Thanks,
Geert-Jan
--
View this message in context: http://www.nabble.com/big-perf-difference-between-solr-server-vs.--SOlrJ-req.process%28solrserver%29-tp14513964p14514441.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: big perf-difference between solr-server vs. SOlrJ req.process(solrserver)
Posted by Yonik Seeley <yo...@apache.org>.
On Dec 27, 2007 9:45 AM, Britske <gb...@gmail.com> wrote:
> I am using SolrJ to communicate with SOLR. My Solr-queries perform within
> range (between 50 ms and 300 ms) by looking at the solr log as ouputted on
> my (windows) commandline.
>
> However I discovered that the following command at all times takes
> significantly longer than the number outputted in the solr-log, (sometimes
> about 400% longer):
It's probably due to stored field retrieval.
The time in the response includes everything except the time to write
the response (since it appears at the beginning). Writing the
response involves reading the stored fields of documents (this was
done to allow one to stream a large number of documents w/o having
them all in memory).
SolrJ's parsing of the response should be a relatively small constant cost.
-Yonik