Re: Huge load and long response times during search

[Patrick Sauts <pa...@gmail.com>]

Try solr.FastLRUCache instead of solr.LRUCache it's the new cache 
gesture for solr 1.4.

And maybe <reopenReaders>true</reopenReaders> in main index section or 
diminish mergefactor
see http://wiki.apache.org/lucene-java/ImproveSearchingSpeed

Tomasz Kępski a écrit :
> Hi,
>
> I'm using SOLR(1.4) to search among about 3,500,000 documents. After 
> the server kernel was updated to 64bit system has started to suffer.
> Our server has 8G of RAM and double Intel Core 2 DUO.
> We used to have average loads around 2-2,5. It was not as good as it 
> should but as long HTTP response times was acceptable we do not care 
> to much ;-)
>
> Since few days avg loads are usually around 6, sometimes goes even to 
> 20. PHP, Mysql and Postgresql based application is rather fine, but 
> when tries to access SOLR it takes ages to load page. In top java 
> process (Jetty) takes 200-250% of CPU, iotop shows that most of the 
> disk operations are done by SOLR threads as well.
>
> When we do shut down Jetty load goes down to 1,5 or even less than 1.
>
> My index has ~12G below is a part of my solrconf.xml:
>
> <query>
>    <maxBooleanClauses>1024</maxBooleanClauses>
>    <filterCache
>      class="solr.LRUCache"
>      size="16384"
>      initialSize="4096"
>      autowarmCount="4096"/>
>    <queryResultCache
>      class="solr.LRUCache"
>      size="16384"
>      initialSize="4096"
>      autowarmCount="1024"/>
>    <documentCache
>      class="solr.LRUCache"
>      size="16384"
>      initialSize="16384"
>      autowarmCount="0"/>
>    <enableLazyFieldLoading>true</enableLazyFieldLoading>
>    <useFilterForSortedQuery>true</useFilterForSortedQuery>
>    <queryResultWindowSize>40</queryResultWindowSize>
>    <queryResultMaxDocsCached>200</queryResultMaxDocsCached>
>    <HashDocSet maxSize="3000" loadFactor="0.75"/>
>    <listener event="newSearcher" class="solr.QuerySenderListener">
>      <arr name="queries">
>        <lst> <str name="q">solr</str> <str name="start">0</str> <str 
> name="rows">10</str> </lst>
>        <lst> <str name="q">solr</str> <str name="sort">price</str> 
> <str name="start">0</str> <str name="rows">10</str> </lst>
>        <lst> <str name="q">solr</str> <str 
> name="sort">rekomendacja</str> <str name="start">0</str> <str 
> name="rows">10</str> </lst>
>        <lst><str name="q">static newSearcher warming query from 
> solrconfig.xml</str></lst>
>      </arr>
>    </listener>
>    <listener event="firstSearcher" class="solr.QuerySenderListener">
>      <arr name="queries">
>        <lst> <str name="q">fast_warm</str> <str name="start">0</str> 
> <str name="rows">10</str> </lst>
>        <lst><str name="q">static firstSearcher warming query from 
> solrconfig.xml</str></lst>
>      </arr>
>    </listener>
>    <useColdSearcher>false</useColdSearcher>
> </query>
>
>  <requestHandler name="dismax" class="solr.SearchHandler" >
>    <lst name="defaults">
>     <str name="defType">dismax</str>
>     <str name="echoParams">explicit</str>
>     <float name="tie">0.01</float>
>     <str name="qf">
>        name^90.0 scategory^450.0 brand^90.0 text^0.01 description^30
>     </str>
>     <str name="pf">
>     </str>
>     <str name="bf">
>     </str>
>     <str name="fl">
>        brand,description,id,name,price,score
>     </str>
>     <str name="mm">
>        4&lt;100% 5&lt;90%
>     </str>
>     <int name="ps">100</int>
>     <str name="q.alt">*:*</str>
>    </lst>
>  </requestHandler>
>
> sample query parameters from log looks like this:
>
> 2009-11-20 21:07:15 org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr path=/select 
> params={spellcheck=true&wt=json&rows=20&json.nl=map&start=520&facet=true&spellcheck.collate=true&fl=id,name,description,preparation,url,shop_id&q=camera&qt=dismax&version=1.3&hl.fl=name,description,atributes,brand,url&facet.field=shop_id&facet.field=brand&hl.fragsize=200&spellcheck.count=5&hl.snippets=3&hl=true} 
> hits=3784 status=0 QTime=83
> 2009-11-20 21:07:15 org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr path=/spellCheckCompRH 
> params={spellcheck=true&wt=json&rows=20&json.nl=map&start=520&facet=true&spellcheck.collate=true&fl=id,name,description,preparation,url,shop_id&q=camera&qt=dismax&version=1.3&hl.fl=name,description,atributes,brand,url&facet.field=shop_id&facet.field=brand&hl.fragsize=200&spellcheck.count=5&hl.snippets=3&hl=true} 
> hits=3784 status=0 QTime=16
>
> And at last the question ;-)
> How to speed up the search?
> Which parameters should I check first to find out what is the bottleneck?
>
> Sorry for verbose entry but I would like to give as clear point of 
> view as possible
>
> Thanks in advance,
> Tom
>