You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by sunnyfr <jo...@gmail.com> on 2008/12/02 18:04:48 UTC
Solr 1.3 - response time very long
Hi,
I tested my old search engine which is sphinx and my new one which solr and
I've got a uge difference of result.
How can I make it faster?
Thanks a lot,
--
View this message in context: http://www.nabble.com/Solr-1.3---response-time-very-long-tp20795134p20795134.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 1.3 - response time very long
Posted by sunnyfr <jo...@gmail.com>.
Thanks a lot guys for your time,
I appreciate it.
I will follow all your advice.
Yonik Seeley wrote:
>
> On Wed, Dec 3, 2008 at 11:49 AM, sunnyfr <jo...@gmail.com> wrote:
>> Sorry the request is more :
>>
> /select?q=text:"svr09\+tutorial"+AND+status_published:1+AND+status_moderated:0+AND+status_personal:0+AND+status_explicit:0+AND+status_private:0+AND+status_deleted:0+AND+status_error:0+AND+status_read
>> or even I tried :
>
> There are a bunch of things you could try to speed things up a bit:
> 1) optimize the index if you haven't
> 2) use a faster response writer with a more compact format (i.e. add
> wt=javabin for a binary format or wt=json for JSON)
> 3) use fl (field list) to restrict the results to only the fields you need
> 4) never use debugQuery to benchmark performance (I don't think you
> actually did, but you did list it in the example dismax URL)
> 5) pull out clauses that match many documents and that are common
> across many queries into filters.
>
> /select?q=text:"svr09\+tutorial"&fq=status_published:1+AND+status_moderated:0+AND+status_personal:0+AND+status_explicit:0+AND+status_private:0+AND+status_deleted:0+AND+status_error:0+AND+status_read
>
> You can also use multiple filter queries for better caching if some of
> the clauses appear in smaller groups or in isolation. If you can give
> more examples, we can tell what the common parts are.
>
> -Yonik
>
>
--
View this message in context: http://www.nabble.com/Solr-1.3---response-time-very-long-tp20795134p20829777.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 1.3 - response time very long
Posted by sunnyfr <jo...@gmail.com>.
right !!!
Yonik Seeley wrote:
>
> On Thu, Dec 4, 2008 at 11:41 AM, sunnyfr <jo...@gmail.com> wrote:
>>
>> Ok thanks a lot,
>>
>> so I can remove all this part
>
> I wouldn't remove them if they are the source of a copyField (with the
> destination being text).
> Simply change to indexed="false" stored="false"
> otherwise you may get an undefined field exception.
>
> -Yonik
>
>> <field name="title" type="text"
>> indexed="true"
>> stored="false" />
>> <field name="description" type="text" indexed="true"
>> stored="false" />
>>
>> <field name="tag1" type="string" indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="tag2" type="string" indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="tag3" type="string" indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="tag4" type="string" indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="tags" type="string" indexed="true"
>> stored="false" omitNorms="true" multiValued="true" termVectors="true" />
>>
>>
>> and just keep ... :
>> <copyField source="title" dest="text"/>
>> <copyField source="title_en" dest="text"/> ..title_es ...
>> <copyField source="description" dest="text"/>
>> <copyField source="description_en" dest="text"/> ..title_en ...
>> <copyField source="tag1" dest="text"/>
>> <copyField source="tag2" dest="text"/>
>> <copyField source="tag3" dest="text"/>
>> <copyField source="tag4" dest="text"/>
>>
>> Just to be sure ... I index, title and description if for exemple i need
>> to
>> boost them sepearetly... bf. title^2 description^1.5
>> Languages one ... need to be index to apply analyzer/stemmer, and boost
>> them
>> differently according to the country.. but I copy them to be searchable.
>>
>> thanks so much for your time .. again and again...
>>
>>
>> Yonik Seeley wrote:
>>>
>>> remove this entry from the example schema unless you need the
>>> timestamp when it was indexed:
>>>
>>> <field name="timestamp" type="date" indexed="true" stored="true"
>>> default="NOW" multiValued="false"/>
>>>
>>> Also, only index fields you really need to search separately.
>>> For example, if the "description" field is also indexed into the
>>> "text" field via copyField, and you only search it via the "text"
>>> field, then don't store or index the description field.
>>>
>>> Retrieving only ids is something that could be optimized in Solr, but
>>> hasn't been done yet.
>>>
>>> -Yonik
>>>
>>> On Thu, Dec 4, 2008 at 11:10 AM, sunnyfr <jo...@gmail.com> wrote:
>>>>
>>>> Hi Yonik,
>>>>
>>>> I will index my data again .... Can you advice me to optimize a lot my
>>>> data
>>>> and tell me if you see something very wrong or bad for the memory,
>>>> according
>>>> to the fact that I just need to show back the ID, that's it.
>>>> But I need to boost some field ... like description ..
>>>> description_country
>>>> according to the country ...
>>>>
>>>> Thanks a lot, I would appreciate it,
>>>>
>>>>
>>>> <fields>
>>>> <field name="id" type="sint"
>>>> indexed="true"
>>>> stored="true" omitNorms="true" />
>>>>
>>>>
>>>> <field name="status_private" type="boolean"
>>>> indexed="true"
>>>> stored="false" omitNorms="true" />
>>>> <field name="status_... 6 more like that>
>>>>
>>>> <field name="duration" type="sint"
>>>> indexed="true"
>>>> stored="false" omitNorms="true" />
>>>> <field name="created" type="date"
>>>> indexed="true"
>>>> stored="false" omitNorms="true" />
>>>> <field name="modified" type="date"
>>>> indexed="true"
>>>> stored="false" omitNorms="true" />
>>>> <field name="rating_binrate" type="sint"
>>>> indexed="true"
>>>> stored="false" omitNorms="true" />
>>>> <field name="user_id" type="sint"
>>>> indexed="true"
>>>> stored="false" omitNorms="true" />
>>>> <field name="country" type="string"
>>>> indexed="true"
>>>> stored="false" omitNorms="true" />
>>>> <field name="language" type="string"
>>>> indexed="true"
>>>> stored="false" omitNorms="true" />
>>>> <field name="creative_type" type="string"
>>>> indexed="true"
>>>> stored="false" omitNorms="true" />
>>>>
>>>>
>>>> <field name="rel_group_ids" type="sint"
>>>> indexed="true"
>>>> stored="false" omitNorms="true" multiValued="true" />
>>>> ... 3more like that
>>>>
>>>>
>>>>
>>>> <field name="rel_featured_user_ids" type="sint"
>>>> indexed="true"
>>>> stored="false" omitNorms="true" multiValued="true" />
>>>> <field name="rel_featured_group_ids" type="sint"
>>>> indexed="true"
>>>> stored="false" omitNorms="true" multiValued="true" />
>>>>
>>>> <field name="stat_views" type="sint"
>>>> indexed="true"
>>>> stored="false" omitNorms="true" />
>>>> ...6more like that ...
>>>>
>>>> <field name="title" type="text"
>>>> indexed="true"
>>>> stored="false" />
>>>> <field name="title_fr" type="text_fr"
>>>> indexed="true"
>>>> stored="false" />
>>>> <field name="title_en" type="text_en"
>>>> indexed="true"
>>>> stored="false" />
>>>> <field name="title_de" type="text_de"
>>>> indexed="true"
>>>> stored="false" />
>>>> <field name="title_es" type="text_es"
>>>> indexed="true"
>>>> stored="false" />
>>>> <field name="title_ru" type="text_ru"
>>>> indexed="true"
>>>> stored="false" />
>>>> <field name="title_pt" type="text_pt"
>>>> indexed="true"
>>>> stored="false" />
>>>> <field name="title_nl" type="text_nl"
>>>> indexed="true"
>>>> stored="false" />
>>>> <field name="title_el" type="text_el"
>>>> indexed="true"
>>>> stored="false" />
>>>> <field name="title_ja" type="text_ja"
>>>> indexed="true"
>>>> stored="false" />
>>>> <field name="title_it" type="text_it"
>>>> indexed="true"
>>>> stored="false" />
>>>>
>>>> <field name="description" type="text"
>>>> indexed="true"
>>>> stored="false" />
>>>> <field name="description_fr" type="text_fr"
>>>> indexed="true"
>>>> stored="false" />
>>>> <field name="description_en" type="text_en"
>>>> indexed="true"
>>>> stored="false" />
>>>> <field name="description_de" type="text_de"
>>>> indexed="true"
>>>> stored="false" />
>>>> <field name="description_es" type="text_es"
>>>> indexed="true"
>>>> stored="false" />
>>>> <field name="description_ru" type="text_ru"
>>>> indexed="true"
>>>> stored="false" />
>>>> <field name="description_pt" type="text_pt"
>>>> indexed="true"
>>>> stored="false" />
>>>> <field name="description_nl" type="text_nl"
>>>> indexed="true"
>>>> stored="false" />
>>>> <field name="description_el" type="text_el"
>>>> indexed="true"
>>>> stored="false" />
>>>> <field name="description_ja" type="text_ja"
>>>> indexed="true"
>>>> stored="false" />
>>>> <field name="description_it" type="text_it"
>>>> indexed="true"
>>>> stored="false" />
>>>>
>>>> <field name="tag1" type="string"
>>>> indexed="true"
>>>> stored="false" omitNorms="true" />
>>>> <field name="tag2" type="string"
>>>> indexed="true"
>>>> stored="false" omitNorms="true" />
>>>> <field name="tag3" type="string"
>>>> indexed="true"
>>>> stored="false" omitNorms="true" />
>>>> <field name="tag4" type="string"
>>>> indexed="true"
>>>> stored="false" omitNorms="true" />
>>>> <field name="tags" type="string"
>>>> indexed="true"
>>>> stored="false" omitNorms="true" multiValued="true" termVectors="true"
>>>> />
>>>> <field name="owner_login" type="string"
>>>> indexed="true"
>>>> stored="false" omitNorms="true" />
>>>>
>>>> <field name="text" type="text" indexed="true" stored="false"
>>>> multiValued="false"/>
>>>> <field name="timestamp" type="date" indexed="true" stored="true"
>>>> default="NOW" multiValued="false"/>
>>>> <field name="spell" type="textSpell" indexed="true" stored="false"
>>>> multiValued="true"/>
>>>> <dynamicField name="random*" type="random" />
>>>>
>>>> </fields>
>>>>
>>>>
>>>>
>>>>
>>>> Yonik Seeley wrote:
>>>>>
>>>>> On Thu, Dec 4, 2008 at 8:13 AM, sunnyfr <jo...@gmail.com> wrote:
>>>>>>
>>>>>> Hi Yonik,
>>>>>>
>>>>>> I've tried everything but it's doesn't change anything, I tried as
>>>>>> well
>>>>>> the
>>>>>> last trunk version but nothing changed.
>>>>>> There is nothings that I can do about the indexation ...maybe I can
>>>>>> optimize
>>>>>> something before searching ???
>>>>>
>>>>> Did you optimize the index (send in the optimize command) after
>>>>> indexing but before searching?
>>>>>
>>>>> curl http://localhost:8983/solr/update?optimize=true
>>>>>
>>>>>> I'm using linux system, apache 5.5, last solr version updated.
>>>>>> Memory : 8G Intel
>>>>>>
>>>>>> Do you think its a lot for the index size 7.6G for 8.5M of document?
>>>>>
>>>>> So it could be due to the index being slightly to big - subtract out
>>>>> memory for Solr and other stuff, and there's not enough left for
>>>>> everything to fully be cached by the OS.
>>>>>
>>>>> You can make it bigger or smaller depending on how you have the schema
>>>>> configured.
>>>>> The example schema isn't necessarily optimized for speed or size - it
>>>>> serves as an example of many field types and operations.
>>>>>
>>>>> Make sure you only index fields you need to search, sort, or facet on.
>>>>> Make sure you only store fields (marked as stored in the schema) that
>>>>> you really need returned in results.
>>>>> The example schema as copyFields and default values that you don't
>>>>> need - hopefully you've removed them.
>>>>>
>>>>> What's your schema, and do you have more examples of URLs you are
>>>>> sending to Solr (all the parameters)?
>>>>>
>>>>> -Yonik
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Solr-1.3---response-time-very-long-tp20795134p20836635.html
>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Solr-1.3---response-time-very-long-tp20795134p20837252.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Solr-1.3---response-time-very-long-tp20795134p20837404.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 1.3 - response time very long
Posted by Yonik Seeley <yo...@apache.org>.
On Thu, Dec 4, 2008 at 11:41 AM, sunnyfr <jo...@gmail.com> wrote:
>
> Ok thanks a lot,
>
> so I can remove all this part
I wouldn't remove them if they are the source of a copyField (with the
destination being text).
Simply change to indexed="false" stored="false"
otherwise you may get an undefined field exception.
-Yonik
> <field name="title" type="text" indexed="true"
> stored="false" />
> <field name="description" type="text" indexed="true"
> stored="false" />
>
> <field name="tag1" type="string" indexed="true"
> stored="false" omitNorms="true" />
> <field name="tag2" type="string" indexed="true"
> stored="false" omitNorms="true" />
> <field name="tag3" type="string" indexed="true"
> stored="false" omitNorms="true" />
> <field name="tag4" type="string" indexed="true"
> stored="false" omitNorms="true" />
> <field name="tags" type="string" indexed="true"
> stored="false" omitNorms="true" multiValued="true" termVectors="true" />
>
>
> and just keep ... :
> <copyField source="title" dest="text"/>
> <copyField source="title_en" dest="text"/> ..title_es ...
> <copyField source="description" dest="text"/>
> <copyField source="description_en" dest="text"/> ..title_en ...
> <copyField source="tag1" dest="text"/>
> <copyField source="tag2" dest="text"/>
> <copyField source="tag3" dest="text"/>
> <copyField source="tag4" dest="text"/>
>
> Just to be sure ... I index, title and description if for exemple i need to
> boost them sepearetly... bf. title^2 description^1.5
> Languages one ... need to be index to apply analyzer/stemmer, and boost them
> differently according to the country.. but I copy them to be searchable.
>
> thanks so much for your time .. again and again...
>
>
> Yonik Seeley wrote:
>>
>> remove this entry from the example schema unless you need the
>> timestamp when it was indexed:
>>
>> <field name="timestamp" type="date" indexed="true" stored="true"
>> default="NOW" multiValued="false"/>
>>
>> Also, only index fields you really need to search separately.
>> For example, if the "description" field is also indexed into the
>> "text" field via copyField, and you only search it via the "text"
>> field, then don't store or index the description field.
>>
>> Retrieving only ids is something that could be optimized in Solr, but
>> hasn't been done yet.
>>
>> -Yonik
>>
>> On Thu, Dec 4, 2008 at 11:10 AM, sunnyfr <jo...@gmail.com> wrote:
>>>
>>> Hi Yonik,
>>>
>>> I will index my data again .... Can you advice me to optimize a lot my
>>> data
>>> and tell me if you see something very wrong or bad for the memory,
>>> according
>>> to the fact that I just need to show back the ID, that's it.
>>> But I need to boost some field ... like description ..
>>> description_country
>>> according to the country ...
>>>
>>> Thanks a lot, I would appreciate it,
>>>
>>>
>>> <fields>
>>> <field name="id" type="sint" indexed="true"
>>> stored="true" omitNorms="true" />
>>>
>>>
>>> <field name="status_private" type="boolean" indexed="true"
>>> stored="false" omitNorms="true" />
>>> <field name="status_... 6 more like that>
>>>
>>> <field name="duration" type="sint" indexed="true"
>>> stored="false" omitNorms="true" />
>>> <field name="created" type="date" indexed="true"
>>> stored="false" omitNorms="true" />
>>> <field name="modified" type="date" indexed="true"
>>> stored="false" omitNorms="true" />
>>> <field name="rating_binrate" type="sint" indexed="true"
>>> stored="false" omitNorms="true" />
>>> <field name="user_id" type="sint" indexed="true"
>>> stored="false" omitNorms="true" />
>>> <field name="country" type="string" indexed="true"
>>> stored="false" omitNorms="true" />
>>> <field name="language" type="string" indexed="true"
>>> stored="false" omitNorms="true" />
>>> <field name="creative_type" type="string" indexed="true"
>>> stored="false" omitNorms="true" />
>>>
>>>
>>> <field name="rel_group_ids" type="sint" indexed="true"
>>> stored="false" omitNorms="true" multiValued="true" />
>>> ... 3more like that
>>>
>>>
>>>
>>> <field name="rel_featured_user_ids" type="sint" indexed="true"
>>> stored="false" omitNorms="true" multiValued="true" />
>>> <field name="rel_featured_group_ids" type="sint" indexed="true"
>>> stored="false" omitNorms="true" multiValued="true" />
>>>
>>> <field name="stat_views" type="sint" indexed="true"
>>> stored="false" omitNorms="true" />
>>> ...6more like that ...
>>>
>>> <field name="title" type="text"
>>> indexed="true"
>>> stored="false" />
>>> <field name="title_fr" type="text_fr"
>>> indexed="true"
>>> stored="false" />
>>> <field name="title_en" type="text_en"
>>> indexed="true"
>>> stored="false" />
>>> <field name="title_de" type="text_de"
>>> indexed="true"
>>> stored="false" />
>>> <field name="title_es" type="text_es"
>>> indexed="true"
>>> stored="false" />
>>> <field name="title_ru" type="text_ru"
>>> indexed="true"
>>> stored="false" />
>>> <field name="title_pt" type="text_pt"
>>> indexed="true"
>>> stored="false" />
>>> <field name="title_nl" type="text_nl"
>>> indexed="true"
>>> stored="false" />
>>> <field name="title_el" type="text_el"
>>> indexed="true"
>>> stored="false" />
>>> <field name="title_ja" type="text_ja"
>>> indexed="true"
>>> stored="false" />
>>> <field name="title_it" type="text_it"
>>> indexed="true"
>>> stored="false" />
>>>
>>> <field name="description" type="text" indexed="true"
>>> stored="false" />
>>> <field name="description_fr" type="text_fr"
>>> indexed="true"
>>> stored="false" />
>>> <field name="description_en" type="text_en"
>>> indexed="true"
>>> stored="false" />
>>> <field name="description_de" type="text_de"
>>> indexed="true"
>>> stored="false" />
>>> <field name="description_es" type="text_es"
>>> indexed="true"
>>> stored="false" />
>>> <field name="description_ru" type="text_ru"
>>> indexed="true"
>>> stored="false" />
>>> <field name="description_pt" type="text_pt"
>>> indexed="true"
>>> stored="false" />
>>> <field name="description_nl" type="text_nl"
>>> indexed="true"
>>> stored="false" />
>>> <field name="description_el" type="text_el"
>>> indexed="true"
>>> stored="false" />
>>> <field name="description_ja" type="text_ja"
>>> indexed="true"
>>> stored="false" />
>>> <field name="description_it" type="text_it"
>>> indexed="true"
>>> stored="false" />
>>>
>>> <field name="tag1" type="string" indexed="true"
>>> stored="false" omitNorms="true" />
>>> <field name="tag2" type="string" indexed="true"
>>> stored="false" omitNorms="true" />
>>> <field name="tag3" type="string" indexed="true"
>>> stored="false" omitNorms="true" />
>>> <field name="tag4" type="string" indexed="true"
>>> stored="false" omitNorms="true" />
>>> <field name="tags" type="string" indexed="true"
>>> stored="false" omitNorms="true" multiValued="true" termVectors="true" />
>>> <field name="owner_login" type="string" indexed="true"
>>> stored="false" omitNorms="true" />
>>>
>>> <field name="text" type="text" indexed="true" stored="false"
>>> multiValued="false"/>
>>> <field name="timestamp" type="date" indexed="true" stored="true"
>>> default="NOW" multiValued="false"/>
>>> <field name="spell" type="textSpell" indexed="true" stored="false"
>>> multiValued="true"/>
>>> <dynamicField name="random*" type="random" />
>>>
>>> </fields>
>>>
>>>
>>>
>>>
>>> Yonik Seeley wrote:
>>>>
>>>> On Thu, Dec 4, 2008 at 8:13 AM, sunnyfr <jo...@gmail.com> wrote:
>>>>>
>>>>> Hi Yonik,
>>>>>
>>>>> I've tried everything but it's doesn't change anything, I tried as well
>>>>> the
>>>>> last trunk version but nothing changed.
>>>>> There is nothings that I can do about the indexation ...maybe I can
>>>>> optimize
>>>>> something before searching ???
>>>>
>>>> Did you optimize the index (send in the optimize command) after
>>>> indexing but before searching?
>>>>
>>>> curl http://localhost:8983/solr/update?optimize=true
>>>>
>>>>> I'm using linux system, apache 5.5, last solr version updated.
>>>>> Memory : 8G Intel
>>>>>
>>>>> Do you think its a lot for the index size 7.6G for 8.5M of document?
>>>>
>>>> So it could be due to the index being slightly to big - subtract out
>>>> memory for Solr and other stuff, and there's not enough left for
>>>> everything to fully be cached by the OS.
>>>>
>>>> You can make it bigger or smaller depending on how you have the schema
>>>> configured.
>>>> The example schema isn't necessarily optimized for speed or size - it
>>>> serves as an example of many field types and operations.
>>>>
>>>> Make sure you only index fields you need to search, sort, or facet on.
>>>> Make sure you only store fields (marked as stored in the schema) that
>>>> you really need returned in results.
>>>> The example schema as copyFields and default values that you don't
>>>> need - hopefully you've removed them.
>>>>
>>>> What's your schema, and do you have more examples of URLs you are
>>>> sending to Solr (all the parameters)?
>>>>
>>>> -Yonik
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Solr-1.3---response-time-very-long-tp20795134p20836635.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Solr-1.3---response-time-very-long-tp20795134p20837252.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
Re: Solr 1.3 - response time very long
Posted by sunnyfr <jo...@gmail.com>.
Ok thanks a lot,
so I can remove all this part
<field name="title" type="text" indexed="true"
stored="false" />
<field name="description" type="text" indexed="true"
stored="false" />
<field name="tag1" type="string" indexed="true"
stored="false" omitNorms="true" />
<field name="tag2" type="string" indexed="true"
stored="false" omitNorms="true" />
<field name="tag3" type="string" indexed="true"
stored="false" omitNorms="true" />
<field name="tag4" type="string" indexed="true"
stored="false" omitNorms="true" />
<field name="tags" type="string" indexed="true"
stored="false" omitNorms="true" multiValued="true" termVectors="true" />
and just keep ... :
<copyField source="title" dest="text"/>
<copyField source="title_en" dest="text"/> ..title_es ...
<copyField source="description" dest="text"/>
<copyField source="description_en" dest="text"/> ..title_en ...
<copyField source="tag1" dest="text"/>
<copyField source="tag2" dest="text"/>
<copyField source="tag3" dest="text"/>
<copyField source="tag4" dest="text"/>
Just to be sure ... I index, title and description if for exemple i need to
boost them sepearetly... bf. title^2 description^1.5
Languages one ... need to be index to apply analyzer/stemmer, and boost them
differently according to the country.. but I copy them to be searchable.
thanks so much for your time .. again and again...
Yonik Seeley wrote:
>
> remove this entry from the example schema unless you need the
> timestamp when it was indexed:
>
> <field name="timestamp" type="date" indexed="true" stored="true"
> default="NOW" multiValued="false"/>
>
> Also, only index fields you really need to search separately.
> For example, if the "description" field is also indexed into the
> "text" field via copyField, and you only search it via the "text"
> field, then don't store or index the description field.
>
> Retrieving only ids is something that could be optimized in Solr, but
> hasn't been done yet.
>
> -Yonik
>
> On Thu, Dec 4, 2008 at 11:10 AM, sunnyfr <jo...@gmail.com> wrote:
>>
>> Hi Yonik,
>>
>> I will index my data again .... Can you advice me to optimize a lot my
>> data
>> and tell me if you see something very wrong or bad for the memory,
>> according
>> to the fact that I just need to show back the ID, that's it.
>> But I need to boost some field ... like description ..
>> description_country
>> according to the country ...
>>
>> Thanks a lot, I would appreciate it,
>>
>>
>> <fields>
>> <field name="id" type="sint" indexed="true"
>> stored="true" omitNorms="true" />
>>
>>
>> <field name="status_private" type="boolean" indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="status_... 6 more like that>
>>
>> <field name="duration" type="sint" indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="created" type="date" indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="modified" type="date" indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="rating_binrate" type="sint" indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="user_id" type="sint" indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="country" type="string" indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="language" type="string" indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="creative_type" type="string" indexed="true"
>> stored="false" omitNorms="true" />
>>
>>
>> <field name="rel_group_ids" type="sint" indexed="true"
>> stored="false" omitNorms="true" multiValued="true" />
>> ... 3more like that
>>
>>
>>
>> <field name="rel_featured_user_ids" type="sint" indexed="true"
>> stored="false" omitNorms="true" multiValued="true" />
>> <field name="rel_featured_group_ids" type="sint" indexed="true"
>> stored="false" omitNorms="true" multiValued="true" />
>>
>> <field name="stat_views" type="sint" indexed="true"
>> stored="false" omitNorms="true" />
>> ...6more like that ...
>>
>> <field name="title" type="text"
>> indexed="true"
>> stored="false" />
>> <field name="title_fr" type="text_fr"
>> indexed="true"
>> stored="false" />
>> <field name="title_en" type="text_en"
>> indexed="true"
>> stored="false" />
>> <field name="title_de" type="text_de"
>> indexed="true"
>> stored="false" />
>> <field name="title_es" type="text_es"
>> indexed="true"
>> stored="false" />
>> <field name="title_ru" type="text_ru"
>> indexed="true"
>> stored="false" />
>> <field name="title_pt" type="text_pt"
>> indexed="true"
>> stored="false" />
>> <field name="title_nl" type="text_nl"
>> indexed="true"
>> stored="false" />
>> <field name="title_el" type="text_el"
>> indexed="true"
>> stored="false" />
>> <field name="title_ja" type="text_ja"
>> indexed="true"
>> stored="false" />
>> <field name="title_it" type="text_it"
>> indexed="true"
>> stored="false" />
>>
>> <field name="description" type="text" indexed="true"
>> stored="false" />
>> <field name="description_fr" type="text_fr"
>> indexed="true"
>> stored="false" />
>> <field name="description_en" type="text_en"
>> indexed="true"
>> stored="false" />
>> <field name="description_de" type="text_de"
>> indexed="true"
>> stored="false" />
>> <field name="description_es" type="text_es"
>> indexed="true"
>> stored="false" />
>> <field name="description_ru" type="text_ru"
>> indexed="true"
>> stored="false" />
>> <field name="description_pt" type="text_pt"
>> indexed="true"
>> stored="false" />
>> <field name="description_nl" type="text_nl"
>> indexed="true"
>> stored="false" />
>> <field name="description_el" type="text_el"
>> indexed="true"
>> stored="false" />
>> <field name="description_ja" type="text_ja"
>> indexed="true"
>> stored="false" />
>> <field name="description_it" type="text_it"
>> indexed="true"
>> stored="false" />
>>
>> <field name="tag1" type="string" indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="tag2" type="string" indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="tag3" type="string" indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="tag4" type="string" indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="tags" type="string" indexed="true"
>> stored="false" omitNorms="true" multiValued="true" termVectors="true" />
>> <field name="owner_login" type="string" indexed="true"
>> stored="false" omitNorms="true" />
>>
>> <field name="text" type="text" indexed="true" stored="false"
>> multiValued="false"/>
>> <field name="timestamp" type="date" indexed="true" stored="true"
>> default="NOW" multiValued="false"/>
>> <field name="spell" type="textSpell" indexed="true" stored="false"
>> multiValued="true"/>
>> <dynamicField name="random*" type="random" />
>>
>> </fields>
>>
>>
>>
>>
>> Yonik Seeley wrote:
>>>
>>> On Thu, Dec 4, 2008 at 8:13 AM, sunnyfr <jo...@gmail.com> wrote:
>>>>
>>>> Hi Yonik,
>>>>
>>>> I've tried everything but it's doesn't change anything, I tried as well
>>>> the
>>>> last trunk version but nothing changed.
>>>> There is nothings that I can do about the indexation ...maybe I can
>>>> optimize
>>>> something before searching ???
>>>
>>> Did you optimize the index (send in the optimize command) after
>>> indexing but before searching?
>>>
>>> curl http://localhost:8983/solr/update?optimize=true
>>>
>>>> I'm using linux system, apache 5.5, last solr version updated.
>>>> Memory : 8G Intel
>>>>
>>>> Do you think its a lot for the index size 7.6G for 8.5M of document?
>>>
>>> So it could be due to the index being slightly to big - subtract out
>>> memory for Solr and other stuff, and there's not enough left for
>>> everything to fully be cached by the OS.
>>>
>>> You can make it bigger or smaller depending on how you have the schema
>>> configured.
>>> The example schema isn't necessarily optimized for speed or size - it
>>> serves as an example of many field types and operations.
>>>
>>> Make sure you only index fields you need to search, sort, or facet on.
>>> Make sure you only store fields (marked as stored in the schema) that
>>> you really need returned in results.
>>> The example schema as copyFields and default values that you don't
>>> need - hopefully you've removed them.
>>>
>>> What's your schema, and do you have more examples of URLs you are
>>> sending to Solr (all the parameters)?
>>>
>>> -Yonik
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Solr-1.3---response-time-very-long-tp20795134p20836635.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Solr-1.3---response-time-very-long-tp20795134p20837252.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 1.3 - response time very long
Posted by Yonik Seeley <yo...@apache.org>.
remove this entry from the example schema unless you need the
timestamp when it was indexed:
<field name="timestamp" type="date" indexed="true" stored="true"
default="NOW" multiValued="false"/>
Also, only index fields you really need to search separately.
For example, if the "description" field is also indexed into the
"text" field via copyField, and you only search it via the "text"
field, then don't store or index the description field.
Retrieving only ids is something that could be optimized in Solr, but
hasn't been done yet.
-Yonik
On Thu, Dec 4, 2008 at 11:10 AM, sunnyfr <jo...@gmail.com> wrote:
>
> Hi Yonik,
>
> I will index my data again .... Can you advice me to optimize a lot my data
> and tell me if you see something very wrong or bad for the memory, according
> to the fact that I just need to show back the ID, that's it.
> But I need to boost some field ... like description .. description_country
> according to the country ...
>
> Thanks a lot, I would appreciate it,
>
>
> <fields>
> <field name="id" type="sint" indexed="true"
> stored="true" omitNorms="true" />
>
>
> <field name="status_private" type="boolean" indexed="true"
> stored="false" omitNorms="true" />
> <field name="status_... 6 more like that>
>
> <field name="duration" type="sint" indexed="true"
> stored="false" omitNorms="true" />
> <field name="created" type="date" indexed="true"
> stored="false" omitNorms="true" />
> <field name="modified" type="date" indexed="true"
> stored="false" omitNorms="true" />
> <field name="rating_binrate" type="sint" indexed="true"
> stored="false" omitNorms="true" />
> <field name="user_id" type="sint" indexed="true"
> stored="false" omitNorms="true" />
> <field name="country" type="string" indexed="true"
> stored="false" omitNorms="true" />
> <field name="language" type="string" indexed="true"
> stored="false" omitNorms="true" />
> <field name="creative_type" type="string" indexed="true"
> stored="false" omitNorms="true" />
>
>
> <field name="rel_group_ids" type="sint" indexed="true"
> stored="false" omitNorms="true" multiValued="true" />
> ... 3more like that
>
>
>
> <field name="rel_featured_user_ids" type="sint" indexed="true"
> stored="false" omitNorms="true" multiValued="true" />
> <field name="rel_featured_group_ids" type="sint" indexed="true"
> stored="false" omitNorms="true" multiValued="true" />
>
> <field name="stat_views" type="sint" indexed="true"
> stored="false" omitNorms="true" />
> ...6more like that ...
>
> <field name="title" type="text" indexed="true"
> stored="false" />
> <field name="title_fr" type="text_fr" indexed="true"
> stored="false" />
> <field name="title_en" type="text_en" indexed="true"
> stored="false" />
> <field name="title_de" type="text_de" indexed="true"
> stored="false" />
> <field name="title_es" type="text_es" indexed="true"
> stored="false" />
> <field name="title_ru" type="text_ru" indexed="true"
> stored="false" />
> <field name="title_pt" type="text_pt" indexed="true"
> stored="false" />
> <field name="title_nl" type="text_nl" indexed="true"
> stored="false" />
> <field name="title_el" type="text_el" indexed="true"
> stored="false" />
> <field name="title_ja" type="text_ja" indexed="true"
> stored="false" />
> <field name="title_it" type="text_it" indexed="true"
> stored="false" />
>
> <field name="description" type="text" indexed="true"
> stored="false" />
> <field name="description_fr" type="text_fr" indexed="true"
> stored="false" />
> <field name="description_en" type="text_en" indexed="true"
> stored="false" />
> <field name="description_de" type="text_de" indexed="true"
> stored="false" />
> <field name="description_es" type="text_es" indexed="true"
> stored="false" />
> <field name="description_ru" type="text_ru" indexed="true"
> stored="false" />
> <field name="description_pt" type="text_pt" indexed="true"
> stored="false" />
> <field name="description_nl" type="text_nl" indexed="true"
> stored="false" />
> <field name="description_el" type="text_el" indexed="true"
> stored="false" />
> <field name="description_ja" type="text_ja" indexed="true"
> stored="false" />
> <field name="description_it" type="text_it" indexed="true"
> stored="false" />
>
> <field name="tag1" type="string" indexed="true"
> stored="false" omitNorms="true" />
> <field name="tag2" type="string" indexed="true"
> stored="false" omitNorms="true" />
> <field name="tag3" type="string" indexed="true"
> stored="false" omitNorms="true" />
> <field name="tag4" type="string" indexed="true"
> stored="false" omitNorms="true" />
> <field name="tags" type="string" indexed="true"
> stored="false" omitNorms="true" multiValued="true" termVectors="true" />
> <field name="owner_login" type="string" indexed="true"
> stored="false" omitNorms="true" />
>
> <field name="text" type="text" indexed="true" stored="false"
> multiValued="false"/>
> <field name="timestamp" type="date" indexed="true" stored="true"
> default="NOW" multiValued="false"/>
> <field name="spell" type="textSpell" indexed="true" stored="false"
> multiValued="true"/>
> <dynamicField name="random*" type="random" />
>
> </fields>
>
>
>
>
> Yonik Seeley wrote:
>>
>> On Thu, Dec 4, 2008 at 8:13 AM, sunnyfr <jo...@gmail.com> wrote:
>>>
>>> Hi Yonik,
>>>
>>> I've tried everything but it's doesn't change anything, I tried as well
>>> the
>>> last trunk version but nothing changed.
>>> There is nothings that I can do about the indexation ...maybe I can
>>> optimize
>>> something before searching ???
>>
>> Did you optimize the index (send in the optimize command) after
>> indexing but before searching?
>>
>> curl http://localhost:8983/solr/update?optimize=true
>>
>>> I'm using linux system, apache 5.5, last solr version updated.
>>> Memory : 8G Intel
>>>
>>> Do you think its a lot for the index size 7.6G for 8.5M of document?
>>
>> So it could be due to the index being slightly to big - subtract out
>> memory for Solr and other stuff, and there's not enough left for
>> everything to fully be cached by the OS.
>>
>> You can make it bigger or smaller depending on how you have the schema
>> configured.
>> The example schema isn't necessarily optimized for speed or size - it
>> serves as an example of many field types and operations.
>>
>> Make sure you only index fields you need to search, sort, or facet on.
>> Make sure you only store fields (marked as stored in the schema) that
>> you really need returned in results.
>> The example schema as copyFields and default values that you don't
>> need - hopefully you've removed them.
>>
>> What's your schema, and do you have more examples of URLs you are
>> sending to Solr (all the parameters)?
>>
>> -Yonik
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Solr-1.3---response-time-very-long-tp20795134p20836635.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
Re: Solr 1.3 - response time very long
Posted by sunnyfr <jo...@gmail.com>.
Hi Yonik,
I will index my data again .... Can you advice me to optimize a lot my data
and tell me if you see something very wrong or bad for the memory, according
to the fact that I just need to show back the ID, that's it.
But I need to boost some field ... like description .. description_country
according to the country ...
Thanks a lot, I would appreciate it,
<fields>
<field name="id" type="sint" indexed="true"
stored="true" omitNorms="true" />
<field name="status_private" type="boolean" indexed="true"
stored="false" omitNorms="true" />
<field name="status_... 6 more like that>
<field name="duration" type="sint" indexed="true"
stored="false" omitNorms="true" />
<field name="created" type="date" indexed="true"
stored="false" omitNorms="true" />
<field name="modified" type="date" indexed="true"
stored="false" omitNorms="true" />
<field name="rating_binrate" type="sint" indexed="true"
stored="false" omitNorms="true" />
<field name="user_id" type="sint" indexed="true"
stored="false" omitNorms="true" />
<field name="country" type="string" indexed="true"
stored="false" omitNorms="true" />
<field name="language" type="string" indexed="true"
stored="false" omitNorms="true" />
<field name="creative_type" type="string" indexed="true"
stored="false" omitNorms="true" />
<field name="rel_group_ids" type="sint" indexed="true"
stored="false" omitNorms="true" multiValued="true" />
... 3more like that
<field name="rel_featured_user_ids" type="sint" indexed="true"
stored="false" omitNorms="true" multiValued="true" />
<field name="rel_featured_group_ids" type="sint" indexed="true"
stored="false" omitNorms="true" multiValued="true" />
<field name="stat_views" type="sint" indexed="true"
stored="false" omitNorms="true" />
...6more like that ...
<field name="title" type="text" indexed="true"
stored="false" />
<field name="title_fr" type="text_fr" indexed="true"
stored="false" />
<field name="title_en" type="text_en" indexed="true"
stored="false" />
<field name="title_de" type="text_de" indexed="true"
stored="false" />
<field name="title_es" type="text_es" indexed="true"
stored="false" />
<field name="title_ru" type="text_ru" indexed="true"
stored="false" />
<field name="title_pt" type="text_pt" indexed="true"
stored="false" />
<field name="title_nl" type="text_nl" indexed="true"
stored="false" />
<field name="title_el" type="text_el" indexed="true"
stored="false" />
<field name="title_ja" type="text_ja" indexed="true"
stored="false" />
<field name="title_it" type="text_it" indexed="true"
stored="false" />
<field name="description" type="text" indexed="true"
stored="false" />
<field name="description_fr" type="text_fr" indexed="true"
stored="false" />
<field name="description_en" type="text_en" indexed="true"
stored="false" />
<field name="description_de" type="text_de" indexed="true"
stored="false" />
<field name="description_es" type="text_es" indexed="true"
stored="false" />
<field name="description_ru" type="text_ru" indexed="true"
stored="false" />
<field name="description_pt" type="text_pt" indexed="true"
stored="false" />
<field name="description_nl" type="text_nl" indexed="true"
stored="false" />
<field name="description_el" type="text_el" indexed="true"
stored="false" />
<field name="description_ja" type="text_ja" indexed="true"
stored="false" />
<field name="description_it" type="text_it" indexed="true"
stored="false" />
<field name="tag1" type="string" indexed="true"
stored="false" omitNorms="true" />
<field name="tag2" type="string" indexed="true"
stored="false" omitNorms="true" />
<field name="tag3" type="string" indexed="true"
stored="false" omitNorms="true" />
<field name="tag4" type="string" indexed="true"
stored="false" omitNorms="true" />
<field name="tags" type="string" indexed="true"
stored="false" omitNorms="true" multiValued="true" termVectors="true" />
<field name="owner_login" type="string" indexed="true"
stored="false" omitNorms="true" />
<field name="text" type="text" indexed="true" stored="false"
multiValued="false"/>
<field name="timestamp" type="date" indexed="true" stored="true"
default="NOW" multiValued="false"/>
<field name="spell" type="textSpell" indexed="true" stored="false"
multiValued="true"/>
<dynamicField name="random*" type="random" />
</fields>
Yonik Seeley wrote:
>
> On Thu, Dec 4, 2008 at 8:13 AM, sunnyfr <jo...@gmail.com> wrote:
>>
>> Hi Yonik,
>>
>> I've tried everything but it's doesn't change anything, I tried as well
>> the
>> last trunk version but nothing changed.
>> There is nothings that I can do about the indexation ...maybe I can
>> optimize
>> something before searching ???
>
> Did you optimize the index (send in the optimize command) after
> indexing but before searching?
>
> curl http://localhost:8983/solr/update?optimize=true
>
>> I'm using linux system, apache 5.5, last solr version updated.
>> Memory : 8G Intel
>>
>> Do you think its a lot for the index size 7.6G for 8.5M of document?
>
> So it could be due to the index being slightly to big - subtract out
> memory for Solr and other stuff, and there's not enough left for
> everything to fully be cached by the OS.
>
> You can make it bigger or smaller depending on how you have the schema
> configured.
> The example schema isn't necessarily optimized for speed or size - it
> serves as an example of many field types and operations.
>
> Make sure you only index fields you need to search, sort, or facet on.
> Make sure you only store fields (marked as stored in the schema) that
> you really need returned in results.
> The example schema as copyFields and default values that you don't
> need - hopefully you've removed them.
>
> What's your schema, and do you have more examples of URLs you are
> sending to Solr (all the parameters)?
>
> -Yonik
>
>
--
View this message in context: http://www.nabble.com/Solr-1.3---response-time-very-long-tp20795134p20836635.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 1.3 - response time very long
Posted by sunnyfr <jo...@gmail.com>.
Huge thanks for your help Yonik,
I optimized the index so I will try to reduce the size ... like I explained
you I stored all language text ...
So I will reduce my stored data.
Cheers... I will let you know :)
Yonik Seeley wrote:
>
> On Thu, Dec 4, 2008 at 8:13 AM, sunnyfr <jo...@gmail.com> wrote:
>>
>> Hi Yonik,
>>
>> I've tried everything but it's doesn't change anything, I tried as well
>> the
>> last trunk version but nothing changed.
>> There is nothings that I can do about the indexation ...maybe I can
>> optimize
>> something before searching ???
>
> Did you optimize the index (send in the optimize command) after
> indexing but before searching?
>
> curl http://localhost:8983/solr/update?optimize=true
>
>> I'm using linux system, apache 5.5, last solr version updated.
>> Memory : 8G Intel
>>
>> Do you think its a lot for the index size 7.6G for 8.5M of document?
>
> So it could be due to the index being slightly to big - subtract out
> memory for Solr and other stuff, and there's not enough left for
> everything to fully be cached by the OS.
>
> You can make it bigger or smaller depending on how you have the schema
> configured.
> The example schema isn't necessarily optimized for speed or size - it
> serves as an example of many field types and operations.
>
> Make sure you only index fields you need to search, sort, or facet on.
> Make sure you only store fields (marked as stored in the schema) that
> you really need returned in results.
> The example schema as copyFields and default values that you don't
> need - hopefully you've removed them.
>
> What's your schema, and do you have more examples of URLs you are
> sending to Solr (all the parameters)?
>
> -Yonik
>
>
--
View this message in context: http://www.nabble.com/Solr-1.3---response-time-very-long-tp20795134p20834935.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 1.3 - response time very long
Posted by Yonik Seeley <yo...@apache.org>.
On Thu, Dec 4, 2008 at 8:13 AM, sunnyfr <jo...@gmail.com> wrote:
>
> Hi Yonik,
>
> I've tried everything but it's doesn't change anything, I tried as well the
> last trunk version but nothing changed.
> There is nothings that I can do about the indexation ...maybe I can optimize
> something before searching ???
Did you optimize the index (send in the optimize command) after
indexing but before searching?
curl http://localhost:8983/solr/update?optimize=true
> I'm using linux system, apache 5.5, last solr version updated.
> Memory : 8G Intel
>
> Do you think its a lot for the index size 7.6G for 8.5M of document?
So it could be due to the index being slightly to big - subtract out
memory for Solr and other stuff, and there's not enough left for
everything to fully be cached by the OS.
You can make it bigger or smaller depending on how you have the schema
configured.
The example schema isn't necessarily optimized for speed or size - it
serves as an example of many field types and operations.
Make sure you only index fields you need to search, sort, or facet on.
Make sure you only store fields (marked as stored in the schema) that
you really need returned in results.
The example schema as copyFields and default values that you don't
need - hopefully you've removed them.
What's your schema, and do you have more examples of URLs you are
sending to Solr (all the parameters)?
-Yonik
Re: Solr 1.3 - response time very long
Posted by sunnyfr <jo...@gmail.com>.
Hi Yonik,
I've tried everything but it's doesn't change anything, I tried as well the
last trunk version but nothing changed.
There is nothings that I can do about the indexation ...maybe I can optimize
something before searching ???
I'm using linux system, apache 5.5, last solr version updated.
Memory : 8G Intel
Do you think its a lot for the index size 7.6G for 8.5M of document?
And idea what can I do ???
Thanks a lot for your time
Yonik Seeley wrote:
>
> On Wed, Dec 3, 2008 at 11:49 AM, sunnyfr <jo...@gmail.com> wrote:
>> Sorry the request is more :
>>
> /select?q=text:"svr09\+tutorial"+AND+status_published:1+AND+status_moderated:0+AND+status_personal:0+AND+status_explicit:0+AND+status_private:0+AND+status_deleted:0+AND+status_error:0+AND+status_read
>> or even I tried :
>
> There are a bunch of things you could try to speed things up a bit:
> 1) optimize the index if you haven't
> 2) use a faster response writer with a more compact format (i.e. add
> wt=javabin for a binary format or wt=json for JSON)
> 3) use fl (field list) to restrict the results to only the fields you need
> 4) never use debugQuery to benchmark performance (I don't think you
> actually did, but you did list it in the example dismax URL)
> 5) pull out clauses that match many documents and that are common
> across many queries into filters.
>
> /select?q=text:"svr09\+tutorial"&fq=status_published:1+AND+status_moderated:0+AND+status_personal:0+AND+status_explicit:0+AND+status_private:0+AND+status_deleted:0+AND+status_error:0+AND+status_read
>
> You can also use multiple filter queries for better caching if some of
> the clauses appear in smaller groups or in isolation. If you can give
> more examples, we can tell what the common parts are.
>
> -Yonik
>
>
--
View this message in context: http://www.nabble.com/Solr-1.3---response-time-very-long-tp20795134p20833091.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 1.3 - response time very long
Posted by Yonik Seeley <yo...@apache.org>.
On Wed, Dec 3, 2008 at 11:49 AM, sunnyfr <jo...@gmail.com> wrote:
> Sorry the request is more :
>
/select?q=text:"svr09\+tutorial"+AND+status_published:1+AND+status_moderated:0+AND+status_personal:0+AND+status_explicit:0+AND+status_private:0+AND+status_deleted:0+AND+status_error:0+AND+status_read
> or even I tried :
There are a bunch of things you could try to speed things up a bit:
1) optimize the index if you haven't
2) use a faster response writer with a more compact format (i.e. add
wt=javabin for a binary format or wt=json for JSON)
3) use fl (field list) to restrict the results to only the fields you need
4) never use debugQuery to benchmark performance (I don't think you
actually did, but you did list it in the example dismax URL)
5) pull out clauses that match many documents and that are common
across many queries into filters.
/select?q=text:"svr09\+tutorial"&fq=status_published:1+AND+status_moderated:0+AND+status_personal:0+AND+status_explicit:0+AND+status_private:0+AND+status_deleted:0+AND+status_error:0+AND+status_read
You can also use multiple filter queries for better caching if some of
the clauses appear in smaller groups or in isolation. If you can give
more examples, we can tell what the common parts are.
-Yonik
Re: Solr 1.3 - response time very long
Posted by sunnyfr <jo...@gmail.com>.
Sorry the request is more :
/select?q=text:"svr09\+tutorial"+AND+status_published:1+AND+status_moderated:0+AND+status_personal:0+AND+status_explicit:0+AND+status_private:0+AND+status_deleted:0+AND+status_error:0+AND+status_read
or even I tried :
select/?qt=dismax&fl=id,score,%20language,title,status_official,stat_views&q=svr09+tutorial&debugQuery=true&qf=title_en+title^1.1+status_official^2.2+status_creative^1.4+description&bf=recip(rord(created),1,10,10)^25+pow(stat_views,0.1)^4
Thanks Matthew
Matthew Runo wrote:
>
> Are you manipulating the query at all between the url like /test/
> selector?cache=0&backend=solr&request=/relevance/search/D and what
> gets sent to Solr? To me, those don't look like solr requests (I could
> be missing something though). I'd be curious to see the actual
> requests to try and let you know why you're getting an error (what
> error is it giving you?).
>
> Thanks for your time!
>
> Matthew Runo
> Software Engineer, Zappos.com
> mruno@zappos.com - 702-943-7833
>
> On Dec 3, 2008, at 1:02 AM, sunnyfr wrote:
>
>>
>> Hi again,
>>
>> In my test, I've maximum response time : 65 sec for an average at 3,
>> So it might come some request which provide error, for exemple in my
>> test
>> for 50 000 requests I've around 30 requests which get back error,
>> that's why
>> the max time response is 65sec.
>>
>> I just don't get why I've this error on some request. like :
>> /test/selector?cache=0&backend=solr&request=/relevance/search/D
>> /test/selector?cache=0&backend=solr&request=/relevance/search/?f+you
>> /test/selector?cache=0&backend=solr&request=/relevance/search/?
>> /test/selector?cache=0&backend=solr&request=/relevance/search/the
>> /test/selector?cache=0&backend=solr&request=/relevance/search/?????
>> ...
>> When I search it manually not by jMeter .. indeed it takes a long
>> time and
>> then it get back ids.
>> What do you think?
>>
>> Thanks a lot for your help.
>>
>>
>> sunnyfr wrote:
>>>
>>> Hi Matthew, Hi Yonik,
>>>
>>> ...sorry for the flag .. didnt want to ...
>>>
>>> Solr 1.3 / Apache 5.5
>>>
>>> Data's directory size : 7.9G
>>> I'm using jMeter to hit http request, I'm sending exactly the same
>>> on solr
>>> and sphinx(mysql) by http either.
>>>
>>> solr
>>> http://test-search.com/test/selector?cache=0&backend=solr&request=/relevance/search/dog
>>> sphinx
>>> http://test-search.com/test/selector?cache=0&backend=mysql&request=/relevance/search/dog
>>>
>>> when threads are more than 4 it's gettting slower, for a big test
>>> during
>>> 40mn with increasing to 100 threads/sec for solr like for sphinx,
>>> at the
>>> end the average for solr is 3sec and for sphinx 1sec.
>>>
>>> solrconfig.xml : http://www.nabble.com/file/p20802690/solrconf.xml
>>> solrconf.xml
>>>
>>> schema.xml:
>>> <fields>
>>> <field name="id" type="sint"
>>> indexed="true"
>>> stored="true" omitNorms="true" />
>>>
>>> <field name="duration" type="sint"
>>> indexed="true"
>>> stored="false" omitNorms="true" />
>>> <field name="created" type="date"
>>> indexed="true"
>>> stored="true" omitNorms="true" />
>>> <field name="modified" type="date"
>>> indexed="true"
>>> stored="false" omitNorms="true" />
>>> <field name="rating_binrate" type="sint"
>>> indexed="true"
>>> stored="true" omitNorms="true" />
>>> <field name="user_id" type="sint"
>>> indexed="true"
>>> stored="false" omitNorms="true" />
>>> <field name="country" type="string"
>>> indexed="true"
>>> stored="false" omitNorms="true" />
>>> <field name="language" type="string"
>>> indexed="true"
>>> stored="true" omitNorms="true" />
>>> ...
>>> <field name="stat_views" type="sint"
>>> indexed="true"
>>> stored="true" omitNorms="true" />
>>> <field name="stat_views_today" type="sint"
>>> indexed="true"
>>> stored="false" omitNorms="true" />
>>> <field name="stat_views_last_week" type="sint"
>>> indexed="true"
>>> stored="false" omitNorms="true" />
>>> <field name="stat_views_last_month" type="sint"
>>> indexed="true"
>>> stored="false" omitNorms="true" />
>>> <field name="stat_comments" type="sint"
>>> indexed="true"
>>> stored="false" omitNorms="true" />
>>> <field name="stat_comments_today" type="sint"
>>> indexed="true"
>>> stored="false" omitNorms="true" />
>>> <field name="stat_comments_last_week" type="sint"
>>> indexed="true"
>>> stored="false" omitNorms="true" />
>>> <field name="stat_comments_last_month" type="sint"
>>> indexed="true"
>>> stored="false" omitNorms="true" />
>>> ...
>>> <field name="title" type="text"
>>> indexed="true" stored="true" />
>>> <field name="title_fr" type="text_fr"
>>> indexed="true" stored="false" />
>>> <field name="title_en" type="text_en"
>>> indexed="true" stored="false" />
>>> <field name="title_de" type="text_de"
>>> indexed="true" stored="false" />
>>> <field name="title_es" type="text_es"
>>> indexed="true" stored="false" />
>>> <field name="title_ru" type="text_ru"
>>> indexed="true" stored="false" />
>>> <field name="title_pt" type="text_pt"
>>> indexed="true" stored="false" />
>>> <field name="title_nl" type="text_nl"
>>> indexed="true" stored="false" />
>>> <field name="title_el" type="text_el"
>>> indexed="true" stored="false" />
>>> <field name="title_ja" type="text_ja"
>>> indexed="true" stored="false" />
>>> <field name="title_it" type="text_it"
>>> indexed="true" stored="false" />
>>>
>>> <field name="description" type="text"
>>> indexed="true"
>>> stored="true" />
>>> <field name="description_fr" type="text_fr"
>>> indexed="true" stored="false" />
>>> <field name="description_en" type="text_en"
>>> indexed="true" stored="false" />
>>> <field name="description_de" type="text_de"
>>> indexed="true" stored="false" />
>>> <field name="description_es" type="text_es"
>>> indexed="true" stored="false" />
>>> <field name="description_ru" type="text_ru"
>>> indexed="true" stored="false" />
>>> <field name="description_pt" type="text_pt"
>>> indexed="true" stored="false" />
>>> <field name="description_nl" type="text_nl"
>>> indexed="true" stored="false" />
>>> <field name="description_el" type="text_el"
>>> indexed="true" stored="false" />
>>> <field name="description_ja" type="text_ja"
>>> indexed="true" stored="false" />
>>> ....
>>>
>>> <field name="text" type="text" indexed="true" stored="false"
>>> multiValued="true"/>
>>> <field name="timestamp" type="date" indexed="true" stored="true"
>>> default="NOW" multiValued="false"/>
>>> <field name="spell" type="textSpell" indexed="true" stored="true"
>>> multiValued="true"/>
>>> <dynamicField name="random*" type="random" />
>>> </fields>
>>>
>>> What would you reckon ???
>>> Thanks a lot,
>>>
>>>
>>>
>>>
>>> Matthew Runo wrote:
>>>>
>>>> Could you provide more information? How big is the index? How are
>>>> you
>>>> searching it? Some examples might help pin down the issue.
>>>>
>>>> How long are the queries taking? How long did they take on Sphinx?
>>>>
>>>> Thanks for your time!
>>>>
>>>> Matthew Runo
>>>> Software Engineer, Zappos.com
>>>> mruno@zappos.com - 702-943-7833
>>>>
>>>> On Dec 2, 2008, at 9:04 AM, sunnyfr wrote:
>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> I tested my old search engine which is sphinx and my new one which
>>>>> solr and
>>>>> I've got a uge difference of result.
>>>>> How can I make it faster?
>>>>>
>>>>> Thanks a lot,
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/Solr-1.3---response-time-very-long-tp20795134p20795134.html
>>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Solr-1.3---response-time-very-long-tp20795134p20809121.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>
>
>
--
View this message in context: http://www.nabble.com/Solr-1.3---response-time-very-long-tp20795134p20816754.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 1.3 - response time very long
Posted by Matthew Runo <mr...@zappos.com>.
Are you manipulating the query at all between the url like /test/
selector?cache=0&backend=solr&request=/relevance/search/D and what
gets sent to Solr? To me, those don't look like solr requests (I could
be missing something though). I'd be curious to see the actual
requests to try and let you know why you're getting an error (what
error is it giving you?).
Thanks for your time!
Matthew Runo
Software Engineer, Zappos.com
mruno@zappos.com - 702-943-7833
On Dec 3, 2008, at 1:02 AM, sunnyfr wrote:
>
> Hi again,
>
> In my test, I've maximum response time : 65 sec for an average at 3,
> So it might come some request which provide error, for exemple in my
> test
> for 50 000 requests I've around 30 requests which get back error,
> that's why
> the max time response is 65sec.
>
> I just don't get why I've this error on some request. like :
> /test/selector?cache=0&backend=solr&request=/relevance/search/D
> /test/selector?cache=0&backend=solr&request=/relevance/search/?f+you
> /test/selector?cache=0&backend=solr&request=/relevance/search/?
> /test/selector?cache=0&backend=solr&request=/relevance/search/the
> /test/selector?cache=0&backend=solr&request=/relevance/search/?????
> ...
> When I search it manually not by jMeter .. indeed it takes a long
> time and
> then it get back ids.
> What do you think?
>
> Thanks a lot for your help.
>
>
> sunnyfr wrote:
>>
>> Hi Matthew, Hi Yonik,
>>
>> ...sorry for the flag .. didnt want to ...
>>
>> Solr 1.3 / Apache 5.5
>>
>> Data's directory size : 7.9G
>> I'm using jMeter to hit http request, I'm sending exactly the same
>> on solr
>> and sphinx(mysql) by http either.
>>
>> solr
>> http://test-search.com/test/selector?cache=0&backend=solr&request=/relevance/search/dog
>> sphinx
>> http://test-search.com/test/selector?cache=0&backend=mysql&request=/relevance/search/dog
>>
>> when threads are more than 4 it's gettting slower, for a big test
>> during
>> 40mn with increasing to 100 threads/sec for solr like for sphinx,
>> at the
>> end the average for solr is 3sec and for sphinx 1sec.
>>
>> solrconfig.xml : http://www.nabble.com/file/p20802690/solrconf.xml
>> solrconf.xml
>>
>> schema.xml:
>> <fields>
>> <field name="id" type="sint"
>> indexed="true"
>> stored="true" omitNorms="true" />
>>
>> <field name="duration" type="sint"
>> indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="created" type="date"
>> indexed="true"
>> stored="true" omitNorms="true" />
>> <field name="modified" type="date"
>> indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="rating_binrate" type="sint"
>> indexed="true"
>> stored="true" omitNorms="true" />
>> <field name="user_id" type="sint"
>> indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="country" type="string"
>> indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="language" type="string"
>> indexed="true"
>> stored="true" omitNorms="true" />
>> ...
>> <field name="stat_views" type="sint"
>> indexed="true"
>> stored="true" omitNorms="true" />
>> <field name="stat_views_today" type="sint"
>> indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="stat_views_last_week" type="sint"
>> indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="stat_views_last_month" type="sint"
>> indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="stat_comments" type="sint"
>> indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="stat_comments_today" type="sint"
>> indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="stat_comments_last_week" type="sint"
>> indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="stat_comments_last_month" type="sint"
>> indexed="true"
>> stored="false" omitNorms="true" />
>> ...
>> <field name="title" type="text"
>> indexed="true" stored="true" />
>> <field name="title_fr" type="text_fr"
>> indexed="true" stored="false" />
>> <field name="title_en" type="text_en"
>> indexed="true" stored="false" />
>> <field name="title_de" type="text_de"
>> indexed="true" stored="false" />
>> <field name="title_es" type="text_es"
>> indexed="true" stored="false" />
>> <field name="title_ru" type="text_ru"
>> indexed="true" stored="false" />
>> <field name="title_pt" type="text_pt"
>> indexed="true" stored="false" />
>> <field name="title_nl" type="text_nl"
>> indexed="true" stored="false" />
>> <field name="title_el" type="text_el"
>> indexed="true" stored="false" />
>> <field name="title_ja" type="text_ja"
>> indexed="true" stored="false" />
>> <field name="title_it" type="text_it"
>> indexed="true" stored="false" />
>>
>> <field name="description" type="text"
>> indexed="true"
>> stored="true" />
>> <field name="description_fr" type="text_fr"
>> indexed="true" stored="false" />
>> <field name="description_en" type="text_en"
>> indexed="true" stored="false" />
>> <field name="description_de" type="text_de"
>> indexed="true" stored="false" />
>> <field name="description_es" type="text_es"
>> indexed="true" stored="false" />
>> <field name="description_ru" type="text_ru"
>> indexed="true" stored="false" />
>> <field name="description_pt" type="text_pt"
>> indexed="true" stored="false" />
>> <field name="description_nl" type="text_nl"
>> indexed="true" stored="false" />
>> <field name="description_el" type="text_el"
>> indexed="true" stored="false" />
>> <field name="description_ja" type="text_ja"
>> indexed="true" stored="false" />
>> ....
>>
>> <field name="text" type="text" indexed="true" stored="false"
>> multiValued="true"/>
>> <field name="timestamp" type="date" indexed="true" stored="true"
>> default="NOW" multiValued="false"/>
>> <field name="spell" type="textSpell" indexed="true" stored="true"
>> multiValued="true"/>
>> <dynamicField name="random*" type="random" />
>> </fields>
>>
>> What would you reckon ???
>> Thanks a lot,
>>
>>
>>
>>
>> Matthew Runo wrote:
>>>
>>> Could you provide more information? How big is the index? How are
>>> you
>>> searching it? Some examples might help pin down the issue.
>>>
>>> How long are the queries taking? How long did they take on Sphinx?
>>>
>>> Thanks for your time!
>>>
>>> Matthew Runo
>>> Software Engineer, Zappos.com
>>> mruno@zappos.com - 702-943-7833
>>>
>>> On Dec 2, 2008, at 9:04 AM, sunnyfr wrote:
>>>
>>>>
>>>> Hi,
>>>>
>>>> I tested my old search engine which is sphinx and my new one which
>>>> solr and
>>>> I've got a uge difference of result.
>>>> How can I make it faster?
>>>>
>>>> Thanks a lot,
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Solr-1.3---response-time-very-long-tp20795134p20795134.html
>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>
>>>
>>>
>>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Solr-1.3---response-time-very-long-tp20795134p20809121.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Solr 1.3 - response time very long
Posted by sunnyfr <jo...@gmail.com>.
this is my error :
Caused by: java.net.SocketException: Unexpected end of file from server
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
It's like it doesn't find data but it takes time to look for it ???
sunnyfr wrote:
>
> Hi again,
>
> In my test, I've maximum response time : 65 sec for an average at 3,
> So it might come some request which provide error, for exemple in my test
> for 50 000 requests I've around 30 requests which get back error, that's
> why the max time response is 65sec.
>
> I just don't get why I've this error on some request. like :
> /test/selector?cache=0&backend=solr&request=/relevance/search/D
> /test/selector?cache=0&backend=solr&request=/relevance/search/?f+you
> /test/selector?cache=0&backend=solr&request=/relevance/search/?
> /test/selector?cache=0&backend=solr&request=/relevance/search/the
> /test/selector?cache=0&backend=solr&request=/relevance/search/?????
> ...
> When I search it manually not by jMeter .. indeed it takes a long time and
> then it get back ids.
> What do you think?
>
> Thanks a lot for your help.
>
>
> sunnyfr wrote:
>>
>> Hi Matthew, Hi Yonik,
>>
>> ...sorry for the flag .. didnt want to ...
>>
>> Solr 1.3 / Apache 5.5
>>
>> Data's directory size : 7.9G
>> I'm using jMeter to hit http request, I'm sending exactly the same on
>> solr and sphinx(mysql) by http either.
>>
>> solr
>> http://test-search.com/test/selector?cache=0&backend=solr&request=/relevance/search/dog
>> sphinx
>> http://test-search.com/test/selector?cache=0&backend=mysql&request=/relevance/search/dog
>>
>> when threads are more than 4 it's gettting slower, for a big test during
>> 40mn with increasing to 100 threads/sec for solr like for sphinx, at the
>> end the average for solr is 3sec and for sphinx 1sec.
>>
>> solrconfig.xml : http://www.nabble.com/file/p20802690/solrconf.xml
>> solrconf.xml
>>
>> schema.xml:
>> <fields>
>> <field name="id" type="sint" indexed="true"
>> stored="true" omitNorms="true" />
>>
>> <field name="duration" type="sint" indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="created" type="date" indexed="true"
>> stored="true" omitNorms="true" />
>> <field name="modified" type="date" indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="rating_binrate" type="sint" indexed="true"
>> stored="true" omitNorms="true" />
>> <field name="user_id" type="sint" indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="country" type="string" indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="language" type="string" indexed="true"
>> stored="true" omitNorms="true" />
>> ...
>> <field name="stat_views" type="sint" indexed="true"
>> stored="true" omitNorms="true" />
>> <field name="stat_views_today" type="sint" indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="stat_views_last_week" type="sint" indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="stat_views_last_month" type="sint" indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="stat_comments" type="sint" indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="stat_comments_today" type="sint" indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="stat_comments_last_week" type="sint" indexed="true"
>> stored="false" omitNorms="true" />
>> <field name="stat_comments_last_month" type="sint" indexed="true"
>> stored="false" omitNorms="true" />
>> ...
>> <field name="title" type="text"
>> indexed="true" stored="true" />
>> <field name="title_fr" type="text_fr"
>> indexed="true" stored="false" />
>> <field name="title_en" type="text_en"
>> indexed="true" stored="false" />
>> <field name="title_de" type="text_de"
>> indexed="true" stored="false" />
>> <field name="title_es" type="text_es"
>> indexed="true" stored="false" />
>> <field name="title_ru" type="text_ru"
>> indexed="true" stored="false" />
>> <field name="title_pt" type="text_pt"
>> indexed="true" stored="false" />
>> <field name="title_nl" type="text_nl"
>> indexed="true" stored="false" />
>> <field name="title_el" type="text_el"
>> indexed="true" stored="false" />
>> <field name="title_ja" type="text_ja"
>> indexed="true" stored="false" />
>> <field name="title_it" type="text_it"
>> indexed="true" stored="false" />
>>
>> <field name="description" type="text" indexed="true"
>> stored="true" />
>> <field name="description_fr" type="text_fr"
>> indexed="true" stored="false" />
>> <field name="description_en" type="text_en"
>> indexed="true" stored="false" />
>> <field name="description_de" type="text_de"
>> indexed="true" stored="false" />
>> <field name="description_es" type="text_es"
>> indexed="true" stored="false" />
>> <field name="description_ru" type="text_ru"
>> indexed="true" stored="false" />
>> <field name="description_pt" type="text_pt"
>> indexed="true" stored="false" />
>> <field name="description_nl" type="text_nl"
>> indexed="true" stored="false" />
>> <field name="description_el" type="text_el"
>> indexed="true" stored="false" />
>> <field name="description_ja" type="text_ja"
>> indexed="true" stored="false" />
>> ....
>>
>> <field name="text" type="text" indexed="true" stored="false"
>> multiValued="true"/>
>> <field name="timestamp" type="date" indexed="true" stored="true"
>> default="NOW" multiValued="false"/>
>> <field name="spell" type="textSpell" indexed="true" stored="true"
>> multiValued="true"/>
>> <dynamicField name="random*" type="random" />
>> </fields>
>>
>> What would you reckon ???
>> Thanks a lot,
>>
>>
>>
>>
>> Matthew Runo wrote:
>>>
>>> Could you provide more information? How big is the index? How are you
>>> searching it? Some examples might help pin down the issue.
>>>
>>> How long are the queries taking? How long did they take on Sphinx?
>>>
>>> Thanks for your time!
>>>
>>> Matthew Runo
>>> Software Engineer, Zappos.com
>>> mruno@zappos.com - 702-943-7833
>>>
>>> On Dec 2, 2008, at 9:04 AM, sunnyfr wrote:
>>>
>>>>
>>>> Hi,
>>>>
>>>> I tested my old search engine which is sphinx and my new one which
>>>> solr and
>>>> I've got a uge difference of result.
>>>> How can I make it faster?
>>>>
>>>> Thanks a lot,
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Solr-1.3---response-time-very-long-tp20795134p20795134.html
>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>
>>>
>>>
>>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Solr-1.3---response-time-very-long-tp20795134p20810804.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 1.3 - response time very long
Posted by sunnyfr <jo...@gmail.com>.
Hi again,
In my test, I've maximum response time : 65 sec for an average at 3,
So it might come some request which provide error, for exemple in my test
for 50 000 requests I've around 30 requests which get back error, that's why
the max time response is 65sec.
I just don't get why I've this error on some request. like :
/test/selector?cache=0&backend=solr&request=/relevance/search/D
/test/selector?cache=0&backend=solr&request=/relevance/search/?f+you
/test/selector?cache=0&backend=solr&request=/relevance/search/?
/test/selector?cache=0&backend=solr&request=/relevance/search/the
/test/selector?cache=0&backend=solr&request=/relevance/search/?????
...
When I search it manually not by jMeter .. indeed it takes a long time and
then it get back ids.
What do you think?
Thanks a lot for your help.
sunnyfr wrote:
>
> Hi Matthew, Hi Yonik,
>
> ...sorry for the flag .. didnt want to ...
>
> Solr 1.3 / Apache 5.5
>
> Data's directory size : 7.9G
> I'm using jMeter to hit http request, I'm sending exactly the same on solr
> and sphinx(mysql) by http either.
>
> solr
> http://test-search.com/test/selector?cache=0&backend=solr&request=/relevance/search/dog
> sphinx
> http://test-search.com/test/selector?cache=0&backend=mysql&request=/relevance/search/dog
>
> when threads are more than 4 it's gettting slower, for a big test during
> 40mn with increasing to 100 threads/sec for solr like for sphinx, at the
> end the average for solr is 3sec and for sphinx 1sec.
>
> solrconfig.xml : http://www.nabble.com/file/p20802690/solrconf.xml
> solrconf.xml
>
> schema.xml:
> <fields>
> <field name="id" type="sint" indexed="true"
> stored="true" omitNorms="true" />
>
> <field name="duration" type="sint" indexed="true"
> stored="false" omitNorms="true" />
> <field name="created" type="date" indexed="true"
> stored="true" omitNorms="true" />
> <field name="modified" type="date" indexed="true"
> stored="false" omitNorms="true" />
> <field name="rating_binrate" type="sint" indexed="true"
> stored="true" omitNorms="true" />
> <field name="user_id" type="sint" indexed="true"
> stored="false" omitNorms="true" />
> <field name="country" type="string" indexed="true"
> stored="false" omitNorms="true" />
> <field name="language" type="string" indexed="true"
> stored="true" omitNorms="true" />
> ...
> <field name="stat_views" type="sint" indexed="true"
> stored="true" omitNorms="true" />
> <field name="stat_views_today" type="sint" indexed="true"
> stored="false" omitNorms="true" />
> <field name="stat_views_last_week" type="sint" indexed="true"
> stored="false" omitNorms="true" />
> <field name="stat_views_last_month" type="sint" indexed="true"
> stored="false" omitNorms="true" />
> <field name="stat_comments" type="sint" indexed="true"
> stored="false" omitNorms="true" />
> <field name="stat_comments_today" type="sint" indexed="true"
> stored="false" omitNorms="true" />
> <field name="stat_comments_last_week" type="sint" indexed="true"
> stored="false" omitNorms="true" />
> <field name="stat_comments_last_month" type="sint" indexed="true"
> stored="false" omitNorms="true" />
> ...
> <field name="title" type="text"
> indexed="true" stored="true" />
> <field name="title_fr" type="text_fr"
> indexed="true" stored="false" />
> <field name="title_en" type="text_en"
> indexed="true" stored="false" />
> <field name="title_de" type="text_de"
> indexed="true" stored="false" />
> <field name="title_es" type="text_es"
> indexed="true" stored="false" />
> <field name="title_ru" type="text_ru"
> indexed="true" stored="false" />
> <field name="title_pt" type="text_pt"
> indexed="true" stored="false" />
> <field name="title_nl" type="text_nl"
> indexed="true" stored="false" />
> <field name="title_el" type="text_el"
> indexed="true" stored="false" />
> <field name="title_ja" type="text_ja"
> indexed="true" stored="false" />
> <field name="title_it" type="text_it"
> indexed="true" stored="false" />
>
> <field name="description" type="text" indexed="true"
> stored="true" />
> <field name="description_fr" type="text_fr"
> indexed="true" stored="false" />
> <field name="description_en" type="text_en"
> indexed="true" stored="false" />
> <field name="description_de" type="text_de"
> indexed="true" stored="false" />
> <field name="description_es" type="text_es"
> indexed="true" stored="false" />
> <field name="description_ru" type="text_ru"
> indexed="true" stored="false" />
> <field name="description_pt" type="text_pt"
> indexed="true" stored="false" />
> <field name="description_nl" type="text_nl"
> indexed="true" stored="false" />
> <field name="description_el" type="text_el"
> indexed="true" stored="false" />
> <field name="description_ja" type="text_ja"
> indexed="true" stored="false" />
> ....
>
> <field name="text" type="text" indexed="true" stored="false"
> multiValued="true"/>
> <field name="timestamp" type="date" indexed="true" stored="true"
> default="NOW" multiValued="false"/>
> <field name="spell" type="textSpell" indexed="true" stored="true"
> multiValued="true"/>
> <dynamicField name="random*" type="random" />
> </fields>
>
> What would you reckon ???
> Thanks a lot,
>
>
>
>
> Matthew Runo wrote:
>>
>> Could you provide more information? How big is the index? How are you
>> searching it? Some examples might help pin down the issue.
>>
>> How long are the queries taking? How long did they take on Sphinx?
>>
>> Thanks for your time!
>>
>> Matthew Runo
>> Software Engineer, Zappos.com
>> mruno@zappos.com - 702-943-7833
>>
>> On Dec 2, 2008, at 9:04 AM, sunnyfr wrote:
>>
>>>
>>> Hi,
>>>
>>> I tested my old search engine which is sphinx and my new one which
>>> solr and
>>> I've got a uge difference of result.
>>> How can I make it faster?
>>>
>>> Thanks a lot,
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Solr-1.3---response-time-very-long-tp20795134p20795134.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Solr-1.3---response-time-very-long-tp20795134p20809121.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 1.3 - response time very long
Posted by sunnyfr <jo...@gmail.com>.
Hi Matthew, Hi Yonik,
...sorry for the flag .. didnt want to ...
Solr 1.3 / Apache 5.5
Data's directory size : 7.9G
I'm using jMeter to hit http request, I'm sending exactly the same on solr
and sphinx(mysql) by http either.
solr
http://test-search.com/test/selector?cache=0&backend=solr&request=/relevance/search/dog
sphinx
http://test-search.com/test/selector?cache=0&backend=mysql&request=/relevance/search/dog
when threads are more than 4 it's gettting slower, for a big test during
40mn with increasing to 100 threads/sec for solr like for sphinx, at the end
the average for solr is 3sec and for sphinx 1sec.
solrconfig.xml : http://www.nabble.com/file/p20802690/solrconf.xml
solrconf.xml
schema.xml:
<fields>
<field name="id" type="sint" indexed="true"
stored="true" omitNorms="true" />
<field name="duration" type="sint" indexed="true"
stored="false" omitNorms="true" />
<field name="created" type="date" indexed="true"
stored="true" omitNorms="true" />
<field name="modified" type="date" indexed="true"
stored="false" omitNorms="true" />
<field name="rating_binrate" type="sint" indexed="true"
stored="true" omitNorms="true" />
<field name="user_id" type="sint" indexed="true"
stored="false" omitNorms="true" />
<field name="country" type="string" indexed="true"
stored="false" omitNorms="true" />
<field name="language" type="string" indexed="true"
stored="true" omitNorms="true" />
...
<field name="stat_views" type="sint" indexed="true"
stored="true" omitNorms="true" />
<field name="stat_views_today" type="sint" indexed="true"
stored="false" omitNorms="true" />
<field name="stat_views_last_week" type="sint" indexed="true"
stored="false" omitNorms="true" />
<field name="stat_views_last_month" type="sint" indexed="true"
stored="false" omitNorms="true" />
<field name="stat_comments" type="sint" indexed="true"
stored="false" omitNorms="true" />
<field name="stat_comments_today" type="sint" indexed="true"
stored="false" omitNorms="true" />
<field name="stat_comments_last_week" type="sint" indexed="true"
stored="false" omitNorms="true" />
<field name="stat_comments_last_month" type="sint" indexed="true"
stored="false" omitNorms="true" />
...
<field name="title" type="text" indexed="true"
stored="true" />
<field name="title_fr" type="text_fr" indexed="true"
stored="false" />
<field name="title_en" type="text_en" indexed="true"
stored="false" />
<field name="title_de" type="text_de" indexed="true"
stored="false" />
<field name="title_es" type="text_es" indexed="true"
stored="false" />
<field name="title_ru" type="text_ru" indexed="true"
stored="false" />
<field name="title_pt" type="text_pt" indexed="true"
stored="false" />
<field name="title_nl" type="text_nl" indexed="true"
stored="false" />
<field name="title_el" type="text_el" indexed="true"
stored="false" />
<field name="title_ja" type="text_ja" indexed="true"
stored="false" />
<field name="title_it" type="text_it" indexed="true"
stored="false" />
<field name="description" type="text" indexed="true"
stored="true" />
<field name="description_fr" type="text_fr" indexed="true"
stored="false" />
<field name="description_en" type="text_en" indexed="true"
stored="false" />
<field name="description_de" type="text_de" indexed="true"
stored="false" />
<field name="description_es" type="text_es" indexed="true"
stored="false" />
<field name="description_ru" type="text_ru" indexed="true"
stored="false" />
<field name="description_pt" type="text_pt" indexed="true"
stored="false" />
<field name="description_nl" type="text_nl" indexed="true"
stored="false" />
<field name="description_el" type="text_el" indexed="true"
stored="false" />
<field name="description_ja" type="text_ja" indexed="true"
stored="false" />
....
<field name="text" type="text" indexed="true" stored="false"
multiValued="true"/>
<field name="timestamp" type="date" indexed="true" stored="true"
default="NOW" multiValued="false"/>
<field name="spell" type="textSpell" indexed="true" stored="true"
multiValued="true"/>
<dynamicField name="random*" type="random" />
</fields>
What would you reckon ???
Thanks a lot,
Matthew Runo wrote:
>
> Could you provide more information? How big is the index? How are you
> searching it? Some examples might help pin down the issue.
>
> How long are the queries taking? How long did they take on Sphinx?
>
> Thanks for your time!
>
> Matthew Runo
> Software Engineer, Zappos.com
> mruno@zappos.com - 702-943-7833
>
> On Dec 2, 2008, at 9:04 AM, sunnyfr wrote:
>
>>
>> Hi,
>>
>> I tested my old search engine which is sphinx and my new one which
>> solr and
>> I've got a uge difference of result.
>> How can I make it faster?
>>
>> Thanks a lot,
>> --
>> View this message in context:
>> http://www.nabble.com/Solr-1.3---response-time-very-long-tp20795134p20795134.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>
>
>
--
View this message in context: http://www.nabble.com/Solr-1.3---response-time-very-long-tp20795134p20802690.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 1.3 - response time very long
Posted by Matthew Runo <mr...@zappos.com>.
Could you provide more information? How big is the index? How are you
searching it? Some examples might help pin down the issue.
How long are the queries taking? How long did they take on Sphinx?
Thanks for your time!
Matthew Runo
Software Engineer, Zappos.com
mruno@zappos.com - 702-943-7833
On Dec 2, 2008, at 9:04 AM, sunnyfr wrote:
>
> Hi,
>
> I tested my old search engine which is sphinx and my new one which
> solr and
> I've got a uge difference of result.
> How can I make it faster?
>
> Thanks a lot,
> --
> View this message in context: http://www.nabble.com/Solr-1.3---response-time-very-long-tp20795134p20795134.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Solr 1.3 - response time very long
Posted by Yonik Seeley <yo...@apache.org>.
On Tue, Dec 2, 2008 at 12:04 PM, sunnyfr <jo...@gmail.com> wrote:
> How can I make it faster?
There's no "-go-faster-please" flag ;-)
Give us the exact URL and we might be able to help figure out what part is slow.
-Yonik