solr indexing slows down after few minutes

[amit <am...@gmail.com>]

Hi 
I am indexing our website using solr. After crawling the website I am adding
the contents to the solr server and committing. My slor server is on 3.6
version.

I observed that initially the indexing is very fast. It added around 33
docs/sec; making 2000 / minute. But then after the 30 mins or so the
performance reduces drastically. Within an hour of indexing it is adding 200
docs/min which is 1/10 times what it used to add.

Here is the details the time and the total docs in slor.
Time	Docs in solr
(min)
3.48	0
3.54	10,000
3.58	16,000
4.03	24,000
4:08	30,000
4:15	38,000
4:18	41,000
4:25	42,230
4:31	43,140
4:38	44,410

This is after making the name field stored="false". Otherwise the same
behavior can be duplicated much before. My doc has 3 fields and the name
field is the extracted text from the webpages. Others are too small
Not sure what is wrong. I am using a win 7 machine dual core and 4 GB ram.

Thanks
Amit




--
View this message in context: http://lucene.472066.n3.nabble.com/solr-indexing-slows-down-after-few-minutes-tp4004337.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr indexing slows down after few minutes

[amit <am...@gmail.com>]

Commit is not too often, it's a batch of 100 records, takes 40 to 60 secs
before another commit.
No I am not indexing with multi threads. It uses a single thread executor.

I have seen steady performance for now after increasing the merge factor
from 10 to 25.
Will have to wait and watch if that reduces the search speed, but so far so
good.

Thanks
Amit

On Thu, Aug 30, 2012 at 10:53 PM, pravesh [via Lucene] <
ml-node+s472066n4004421h99@n3.nabble.com> wrote:

> Did you checked wiki:
> http://wiki.apache.org/lucene-java/ImproveIndexingSpeed
>
> Do you commit often? Do you index with multiple threads? Also try
> experimenting with various available MergePolicies introduced from SOLR 3.4
> onwards
>
> Thanx
> Pravesh
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>
> http://lucene.472066.n3.nabble.com/solr-indexing-slows-down-after-few-minutes-tp4004337p4004421.html
>  To unsubscribe from solr indexing slows down after few minutes, click
> here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4004337&code=YW1pdC5tYWxsaWtAZ21haWwuY29tfDQwMDQzMzd8LTk5Njc5OTA3NA==>
> .
> NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://lucene.472066.n3.nabble.com/solr-indexing-slows-down-after-few-minutes-tp4004337p4005864.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr indexing slows down after few minutes

[pravesh <su...@yahoo.com>]

Did you checked wiki: 
http://wiki.apache.org/lucene-java/ImproveIndexingSpeed
http://wiki.apache.org/lucene-java/ImproveIndexingSpeed 

Do you commit often? Do you index with multiple threads? Also try
experimenting with various available MergePolicies introduced from SOLR 3.4
onwards

Thanx
Pravesh



--
View this message in context: http://lucene.472066.n3.nabble.com/solr-indexing-slows-down-after-few-minutes-tp4004337p4004421.html
Sent from the Solr - User mailing list archive at Nabble.com.