You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Sujatha Arun <su...@gmail.com> on 2012/09/24 13:26:11 UTC
Performance Degradation on Migrating from 1.3 to solr 3.6.1
Hi,
On migrating from 1.3 to 3.6.1 , I see the query performance degrading by
nearly 2 times for all types of query. Indexing performance slight
degradation over 1.3 For Indexing we use our custom scripts that post xml
over HTTP.
Any thing that I might have missed . I am thinking that this might be due
to new Tiered MP over LogByteSize creating more segment files and hence
more the Query latency .We are using Compound Files in 1.3 and I have set
this to true even in 3.6.1 ,but results in more segement files
On optimizing the query response time improved beyond 1.3 .So could it be
the MP or am i missing something here . Do let me know
Please find attached the solrconfig.xml
Regards
Sujatha
Re: Performance Degradation on Migrating from 1.3 to solr 3.6.1
Posted by Sujatha Arun <su...@gmail.com>.
Hi ,
Please comment on whether I should consider to move to the old Logbytesize
MP on moving to 3.6.1 from 1.3 ,as I see improvements in query performance
on optimization.
Just to mention we have a lot of indexes in multi cores as well as multiple
webapps and that's the reason we went for CFS in 1.3 to avoid the "too
many open file issue which we had encountered.
Regards
Sujatha
On Tue, Sep 25, 2012 at 9:55 AM, Sujatha Arun <su...@gmail.com> wrote:
> Any comments on this?
>
>
>
> On Mon, Sep 24, 2012 at 10:28 PM, Sujatha Arun <su...@gmail.com>wrote:
>
>> Thanks Jack.
>>
>> so Qtime = Sum of all prepare components + sum of all process components
>> - Debug comp process/prepare time
>>
>> In 3.6.1 the process part of Query component for the following query
>> seems to take 8 times more time? anything missing? For most queries the
>> process part of the Querycomponent seem to take more time in 3.6.1
>>
>>
>> This is *3.6.1 *
>>
>> <response>
>> <lst name="responseHeader"><int name="status">0</int>
>> *<int name="QTime">33</int>*
>> <lst name="params">
>> <str name="debugQuery">on</str>
>> <str name="indent">on</str><str name="start">0</str>
>> *<str name="q">differential AND equations AND has AND one AND
>> solution</str>*
>> <str name="rows">10</str><str name="version">2.2</str></lst></lst>
>>
>> *Debug Output*
>> *
>> *
>> <str name="QParser">LuceneQParser</str>
>> <lst name="timing">
>> <double name="time">33.0</double>
>> <lst name="prepare">
>>
>> <double name="time">3.0</double>
>> *<lst name="org.apache.solr.handler.component.QueryComponent"><double
>> name="time">3.0</double></lst>*<lst
>> name="org.apache.solr.handler.component.FacetComponent"><double
>> name="time">0.0</double></lst><lst
>> name="org.apache.solr.handler.component.MoreLikeThisComponent"><double
>> name="time">0.0</double></lst><lst
>> name="org.apache.solr.handler.component.HighlightComponent"><double
>> name="time">0.0</double></lst><lst
>> name="org.apache.solr.handler.component.StatsComponent"><double
>> name="time">0.0</double></lst><lst
>> name="org.apache.solr.handler.component.DebugComponent"><double
>> name="time">0.0</double>
>>
>> </lst></lst><lst name="process"><double name="time">30.0</double>
>> *<lst name="org.apache.solr.handler.component.QueryComponent"><double
>> name="time">26.0</double></lst>*
>> <lst name="org.apache.solr.handler.component.FacetComponent"><double
>> name="time">0.0</double></lst><lst
>> name="org.apache.solr.handler.component.MoreLikeThisComponent"><double
>> name="time">0.0</double></lst><lst
>> name="org.apache.solr.handler.component.HighlightComponent"><double
>> name="time">0.0</double></lst><lst
>> name="org.apache.solr.handler.component.StatsComponent"><double
>> name="time">0.0</double></lst><lst
>> name="org.apache.solr.handler.component.DebugComponent"><double
>> name="time">4.0</double></lst>
>>
>> *Same query in solr 1.3*
>> *
>> *
>> <lst name="responseHeader">
>> <int name="status">0</int>
>> *<int name="QTime">6</int>*
>> <lst name="params">
>> <str name="debugQuery">on</str><str name="indent">on</str>
>> <str name="start">0</str><str name="q">differential AND equations AND has
>> AND one AND solution</str>
>> <str name="rows">10</str><str name="version">2.2</str>
>>
>> Debug Info
>>
>> <lst name="timing">
>> <double name="time">6.0</double>
>> <lst name="prepare">
>> <double name="time">1.0</double>
>> *<lst name="org.apache.solr.handler.component.QueryComponent"><double
>> name="time">1.0</double></lst>*
>> <lst name="org.apache.solr.handler.component.FacetComponent"><double
>> name="time">0.0</double></lst><lst
>> name="org.apache.solr.handler.component.MoreLikeThisComponent"><double
>> name="time">0.0</double></lst><lst
>> name="org.apache.solr.handler.component.HighlightComponent"><double
>> name="time">0.0</double></lst><lst
>> name="org.apache.solr.handler.component.DebugComponent"><double
>> name="time">0.0</double></lst></lst>
>>
>> <lst name="process">
>> <double name="time">5.0</double>
>> *<lst name="org.apache.solr.handler.component.QueryComponent"><double
>> name="time">3.0</double></lst>*
>> <lst name="org.apache.solr.handler.component.FacetComponent"><double
>> name="time">0.0</double></lst><lst
>> name="org.apache.solr.handler.component.MoreLikeThisComponent"><double
>> name="time">0.0</double></lst><lst
>> name="org.apache.solr.handler.component.HighlightComponent"><double
>> name="time">0.0</double></lst><lst
>> name="org.apache.solr.handler.component.DebugComponent"><double
>> name="time">2.0</double>
>>
>>
>>
>> *On Mon, Sep 24, 2012 at 7:35 PM, Jack Krupansky <jack@basetechnology.com
>> > wrote:*
>>
>>> Run a query on both old and new with &debugQuery=true on your query
>>> request and look at the component timings for possible insight.
>>>
>>> -- Jack Krupansky
>>>
>>> From: Sujatha Arun
>>> Sent: Monday, September 24, 2012 7:26 AM
>>> To: solr-user@lucene.apache.org
>>> Subject: Performance Degradation on Migrating from 1.3 to solr 3.6.1
>>>
>>> Hi,
>>>
>>> On migrating from 1.3 to 3.6.1 , I see the query performance degrading
>>> by nearly 2 times for all types of query. Indexing performance slight
>>> degradation over 1.3 For Indexing we use our custom scripts that post xml
>>> over HTTP.
>>>
>>> Any thing that I might have missed . I am thinking that this might be
>>> due to new Tiered MP over LogByteSize creating more segment files and hence
>>> more the Query latency .We are using Compound Files in 1.3 and I have set
>>> this to true even in 3.6.1 ,but results in more segement files
>>>
>>> On optimizing the query response time improved beyond 1.3 .So could it
>>> be the MP or am i missing something here . Do let me know
>>>
>>> Please find attached the solrconfig.xml
>>>
>>> Regards
>>> Sujatha
>>>
>>
>>
>
Re: Performance Degradation on Migrating from 1.3 to solr 3.6.1
Posted by Sujatha Arun <su...@gmail.com>.
Any comments on this?
On Mon, Sep 24, 2012 at 10:28 PM, Sujatha Arun <su...@gmail.com> wrote:
> Thanks Jack.
>
> so Qtime = Sum of all prepare components + sum of all process components -
> Debug comp process/prepare time
>
> In 3.6.1 the process part of Query component for the following query seems
> to take 8 times more time? anything missing? For most queries the process
> part of the Querycomponent seem to take more time in 3.6.1
>
>
> This is *3.6.1 *
>
> <response>
> <lst name="responseHeader"><int name="status">0</int>
> *<int name="QTime">33</int>*
> <lst name="params">
> <str name="debugQuery">on</str>
> <str name="indent">on</str><str name="start">0</str>
> *<str name="q">differential AND equations AND has AND one AND
> solution</str>*
> <str name="rows">10</str><str name="version">2.2</str></lst></lst>
>
> *Debug Output*
> *
> *
> <str name="QParser">LuceneQParser</str>
> <lst name="timing">
> <double name="time">33.0</double>
> <lst name="prepare">
>
> <double name="time">3.0</double>
> *<lst name="org.apache.solr.handler.component.QueryComponent"><double
> name="time">3.0</double></lst>*<lst
> name="org.apache.solr.handler.component.FacetComponent"><double
> name="time">0.0</double></lst><lst
> name="org.apache.solr.handler.component.MoreLikeThisComponent"><double
> name="time">0.0</double></lst><lst
> name="org.apache.solr.handler.component.HighlightComponent"><double
> name="time">0.0</double></lst><lst
> name="org.apache.solr.handler.component.StatsComponent"><double
> name="time">0.0</double></lst><lst
> name="org.apache.solr.handler.component.DebugComponent"><double
> name="time">0.0</double>
>
> </lst></lst><lst name="process"><double name="time">30.0</double>
> *<lst name="org.apache.solr.handler.component.QueryComponent"><double
> name="time">26.0</double></lst>*
> <lst name="org.apache.solr.handler.component.FacetComponent"><double
> name="time">0.0</double></lst><lst
> name="org.apache.solr.handler.component.MoreLikeThisComponent"><double
> name="time">0.0</double></lst><lst
> name="org.apache.solr.handler.component.HighlightComponent"><double
> name="time">0.0</double></lst><lst
> name="org.apache.solr.handler.component.StatsComponent"><double
> name="time">0.0</double></lst><lst
> name="org.apache.solr.handler.component.DebugComponent"><double
> name="time">4.0</double></lst>
>
> *Same query in solr 1.3*
> *
> *
> <lst name="responseHeader">
> <int name="status">0</int>
> *<int name="QTime">6</int>*
> <lst name="params">
> <str name="debugQuery">on</str><str name="indent">on</str>
> <str name="start">0</str><str name="q">differential AND equations AND has
> AND one AND solution</str>
> <str name="rows">10</str><str name="version">2.2</str>
>
> Debug Info
>
> <lst name="timing">
> <double name="time">6.0</double>
> <lst name="prepare">
> <double name="time">1.0</double>
> *<lst name="org.apache.solr.handler.component.QueryComponent"><double
> name="time">1.0</double></lst>*
> <lst name="org.apache.solr.handler.component.FacetComponent"><double
> name="time">0.0</double></lst><lst
> name="org.apache.solr.handler.component.MoreLikeThisComponent"><double
> name="time">0.0</double></lst><lst
> name="org.apache.solr.handler.component.HighlightComponent"><double
> name="time">0.0</double></lst><lst
> name="org.apache.solr.handler.component.DebugComponent"><double
> name="time">0.0</double></lst></lst>
>
> <lst name="process">
> <double name="time">5.0</double>
> *<lst name="org.apache.solr.handler.component.QueryComponent"><double
> name="time">3.0</double></lst>*
> <lst name="org.apache.solr.handler.component.FacetComponent"><double
> name="time">0.0</double></lst><lst
> name="org.apache.solr.handler.component.MoreLikeThisComponent"><double
> name="time">0.0</double></lst><lst
> name="org.apache.solr.handler.component.HighlightComponent"><double
> name="time">0.0</double></lst><lst
> name="org.apache.solr.handler.component.DebugComponent"><double
> name="time">2.0</double>
>
>
>
> *On Mon, Sep 24, 2012 at 7:35 PM, Jack Krupansky <ja...@basetechnology.com>wrote:
> *
>
>> Run a query on both old and new with &debugQuery=true on your query
>> request and look at the component timings for possible insight.
>>
>> -- Jack Krupansky
>>
>> From: Sujatha Arun
>> Sent: Monday, September 24, 2012 7:26 AM
>> To: solr-user@lucene.apache.org
>> Subject: Performance Degradation on Migrating from 1.3 to solr 3.6.1
>>
>> Hi,
>>
>> On migrating from 1.3 to 3.6.1 , I see the query performance degrading
>> by nearly 2 times for all types of query. Indexing performance slight
>> degradation over 1.3 For Indexing we use our custom scripts that post xml
>> over HTTP.
>>
>> Any thing that I might have missed . I am thinking that this might be due
>> to new Tiered MP over LogByteSize creating more segment files and hence
>> more the Query latency .We are using Compound Files in 1.3 and I have set
>> this to true even in 3.6.1 ,but results in more segement files
>>
>> On optimizing the query response time improved beyond 1.3 .So could it
>> be the MP or am i missing something here . Do let me know
>>
>> Please find attached the solrconfig.xml
>>
>> Regards
>> Sujatha
>>
>
>
Re: Performance Degradation on Migrating from 1.3 to solr 3.6.1
Posted by Sujatha Arun <su...@gmail.com>.
Thanks Jack.
so Qtime = Sum of all prepare components + sum of all process components -
Debug comp process/prepare time
In 3.6.1 the process part of Query component for the following query seems
to take 8 times more time? anything missing? For most queries the process
part of the Querycomponent seem to take more time in 3.6.1
This is *3.6.1 *
<response>
<lst name="responseHeader"><int name="status">0</int>
*<int name="QTime">33</int>*
<lst name="params">
<str name="debugQuery">on</str>
<str name="indent">on</str><str name="start">0</str>
*<str name="q">differential AND equations AND has AND one AND solution</str>
*
<str name="rows">10</str><str name="version">2.2</str></lst></lst>
*Debug Output*
*
*
<str name="QParser">LuceneQParser</str>
<lst name="timing">
<double name="time">33.0</double>
<lst name="prepare">
<double name="time">3.0</double>
*<lst name="org.apache.solr.handler.component.QueryComponent"><double
name="time">3.0</double></lst>*<lst
name="org.apache.solr.handler.component.FacetComponent"><double
name="time">0.0</double></lst><lst
name="org.apache.solr.handler.component.MoreLikeThisComponent"><double
name="time">0.0</double></lst><lst
name="org.apache.solr.handler.component.HighlightComponent"><double
name="time">0.0</double></lst><lst
name="org.apache.solr.handler.component.StatsComponent"><double
name="time">0.0</double></lst><lst
name="org.apache.solr.handler.component.DebugComponent"><double
name="time">0.0</double>
</lst></lst><lst name="process"><double name="time">30.0</double>
*<lst name="org.apache.solr.handler.component.QueryComponent"><double
name="time">26.0</double></lst>*
<lst name="org.apache.solr.handler.component.FacetComponent"><double
name="time">0.0</double></lst><lst
name="org.apache.solr.handler.component.MoreLikeThisComponent"><double
name="time">0.0</double></lst><lst
name="org.apache.solr.handler.component.HighlightComponent"><double
name="time">0.0</double></lst><lst
name="org.apache.solr.handler.component.StatsComponent"><double
name="time">0.0</double></lst><lst
name="org.apache.solr.handler.component.DebugComponent"><double
name="time">4.0</double></lst>
*Same query in solr 1.3*
*
*
<lst name="responseHeader">
<int name="status">0</int>
*<int name="QTime">6</int>*
<lst name="params">
<str name="debugQuery">on</str><str name="indent">on</str>
<str name="start">0</str><str name="q">differential AND equations AND has
AND one AND solution</str>
<str name="rows">10</str><str name="version">2.2</str>
Debug Info
<lst name="timing">
<double name="time">6.0</double>
<lst name="prepare">
<double name="time">1.0</double>
*<lst name="org.apache.solr.handler.component.QueryComponent"><double
name="time">1.0</double></lst>*
<lst name="org.apache.solr.handler.component.FacetComponent"><double
name="time">0.0</double></lst><lst
name="org.apache.solr.handler.component.MoreLikeThisComponent"><double
name="time">0.0</double></lst><lst
name="org.apache.solr.handler.component.HighlightComponent"><double
name="time">0.0</double></lst><lst
name="org.apache.solr.handler.component.DebugComponent"><double
name="time">0.0</double></lst></lst>
<lst name="process">
<double name="time">5.0</double>
*<lst name="org.apache.solr.handler.component.QueryComponent"><double
name="time">3.0</double></lst>*
<lst name="org.apache.solr.handler.component.FacetComponent"><double
name="time">0.0</double></lst><lst
name="org.apache.solr.handler.component.MoreLikeThisComponent"><double
name="time">0.0</double></lst><lst
name="org.apache.solr.handler.component.HighlightComponent"><double
name="time">0.0</double></lst><lst
name="org.apache.solr.handler.component.DebugComponent"><double
name="time">2.0</double>
*On Mon, Sep 24, 2012 at 7:35 PM, Jack Krupansky <ja...@basetechnology.com>wrote:
*
> Run a query on both old and new with &debugQuery=true on your query
> request and look at the component timings for possible insight.
>
> -- Jack Krupansky
>
> From: Sujatha Arun
> Sent: Monday, September 24, 2012 7:26 AM
> To: solr-user@lucene.apache.org
> Subject: Performance Degradation on Migrating from 1.3 to solr 3.6.1
>
> Hi,
>
> On migrating from 1.3 to 3.6.1 , I see the query performance degrading by
> nearly 2 times for all types of query. Indexing performance slight
> degradation over 1.3 For Indexing we use our custom scripts that post xml
> over HTTP.
>
> Any thing that I might have missed . I am thinking that this might be due
> to new Tiered MP over LogByteSize creating more segment files and hence
> more the Query latency .We are using Compound Files in 1.3 and I have set
> this to true even in 3.6.1 ,but results in more segement files
>
> On optimizing the query response time improved beyond 1.3 .So could it be
> the MP or am i missing something here . Do let me know
>
> Please find attached the solrconfig.xml
>
> Regards
> Sujatha
>
Re: Performance Degradation on Migrating from 1.3 to solr 3.6.1
Posted by Jack Krupansky <ja...@basetechnology.com>.
Run a query on both old and new with &debugQuery=true on your query request and look at the component timings for possible insight.
-- Jack Krupansky
From: Sujatha Arun
Sent: Monday, September 24, 2012 7:26 AM
To: solr-user@lucene.apache.org
Subject: Performance Degradation on Migrating from 1.3 to solr 3.6.1
Hi,
On migrating from 1.3 to 3.6.1 , I see the query performance degrading by nearly 2 times for all types of query. Indexing performance slight degradation over 1.3 For Indexing we use our custom scripts that post xml over HTTP.
Any thing that I might have missed . I am thinking that this might be due to new Tiered MP over LogByteSize creating more segment files and hence more the Query latency .We are using Compound Files in 1.3 and I have set this to true even in 3.6.1 ,but results in more segement files
On optimizing the query response time improved beyond 1.3 .So could it be the MP or am i missing something here . Do let me know
Please find attached the solrconfig.xml
Regards
Sujatha