Solr Delta Import Issue

[~$alpha` <la...@gmail.com>]

<http://lucene.472066.n3.nabble.com/file/t482390/1.png> 

SOLR VERSION 6.0.0

As seen in the image, There is a spike which can be observed at every 2
hours.
i.e whenever delta import runs

 1. Response time doubles
 2. CPU load average doubles and if it runs to near to peal hour than it
goes even to 10 times.
 

**My Cache Settings are**

     1. <filterCache class="solr.FastLRUCache"
                     size="5000"
                     initialSize="512"
                     autowarmCount="128"/>
    
        <queryResultCache class="solr.FastLRUCache"
                         size="10000"
                         initialSize="512"
                         autowarmCount="128"/>
    
         <documentCache class="solr.FastLRUCache"
                       size="100000"
                       initialSize="10000"
                       autowarmCount="0"/>
    
          <cache name="perSegFilter"
          class="solr.search.LRUCache"
          size="10"
          initialSize="0"
          autowarmCount="10"
          regenerator="solr.NoOpRegenerator" />
    
        <enableLazyFieldLoading>true</enableLazyFieldLoading>
    
        <queryResultWindowSize>20</queryResultWindowSize>
        <queryResultMaxDocsCached>200</queryResultMaxDocsCached>
        <useColdSearcher>false</useColdSearcher>

Most of my queries are CPU centric as they involved lots of IN queries and
NOT IN Queries. Also, have if cond for scoring. Assuming my queries will
continue to be CPU centric.

**Help**
What Am I doing wrong as my delta import causing so much high response?
Index size : 2GB
Servers: 4
Serving: 1Lac per hour
Also, Delta update results in 2lac record update out of 10lac as one of the
fields of solr (last login) changed frequently.
  






--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr Delta Import Issue

[Shawn Heisey <ap...@elyograg.org>]

On 11/27/2018 10:32 AM, ~$alpha` wrote:
> <http://lucene.472066.n3.nabble.com/file/t482390/1.png>
>
> SOLR VERSION 6.0.0
>
> As seen in the image, There is a spike which can be observed at every 2
> hours.
> i.e whenever delta import runs
>
>   1. Response time doubles
>   2. CPU load average doubles and if it runs to near to peal hour than it
> goes even to 10 times.

Indexing puts a lot of load on a Solr server.  It tends to cause 
disruption to queries running on the same machine, especially if 
available memory is low enough to make the OS disk cache ineffective.

If you're running in SolrCloud mode on version 6, every server does both 
indexing and queries, and there's really no way to change that.  Version 
7 has some additional options, especially in the latest version, 7.5.0.

> **My Cache Settings are**
>
>       1. <filterCache class="solr.FastLRUCache"
>                       size="5000"
>                       initialSize="512"
>                       autowarmCount="128"/>

An autowarmCount of 128 means that every time you do a commit operation, 
Solr is going to execute up to 128 filters (depending on how many are 
actually in the cache) on the new index.  If each one takes 100 
milliseconds, that's nearly 13 seconds of straight filter queries being 
executed on your index, which is probably going to impact performance.  
If each filter takes longer than 100 milliseconds to execute, the amount 
of time and system resources required to execute this warming could be 
REALLY high.  I notice you have the same autowarmCount on your 
queryResultCache.  The same thing goes for that cache -- those queries 
will be re-executed on the new index every time you do a commit that 
opens a new searcher.

> Most of my queries are CPU centric as they involved lots of IN queries and
> NOT IN Queries. Also, have if cond for scoring. Assuming my queries will
> continue to be CPU centric.

What evidence do you have that the queries are CPU-centric?  I'm looking 
for something very specific.

Your mention of load average says to me that you're probably not running 
on a Windows system.  Which I think means you could gather the 
screenshot described here:

https://wiki.apache.org/solr/SolrPerformanceProblems#Process_listing_on_POSIX_operating_systems

Can you gather that and make it available with a file sharing website?  
It would be best if you could do it while the import is running and 
performance goes down.

> **Help**
> What Am I doing wrong as my delta import causing so much high response?
> Index size : 2GB
> Servers: 4
> Serving: 1Lac per hour
> Also, Delta update results in 2lac record update out of 10lac as one of the
> fields of solr (last login) changed frequently.

How long does it take to import those 200000 documents? What are your 
autoCommit and autoSoftCommit settings?  Do you know how long it is 
taking to warm each of your Solr caches?  You can get that information 
in the admin UI, under Plugins/Stats.

Thanks,
Shawn

Re: Solr Delta Import Issue

[~$alpha` <la...@gmail.com>]

Time for import - 5-6 minutes
Warmup time - 40seconds

autoCommit and autoSoftCommit setting both disabled and We fire commit only
after the import is completed.

I have some more doubt 
1.  In case of master-slave, is auto warm strategy is available for slaves 

2. Also should I also have a limit on max connection to make server Solr
server won't go down and if yes how to choose that max no. ? 



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html