You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Peter Karich <pe...@yahoo.de> on 2010/07/29 13:02:29 UTC
Solr Indexing slows down
Hi,
I am indexing a solr 1.4.0 core and commiting gets slower and slower.
Starting from 3-5 seconds for ~200 documents and ending with over 60
seconds after 800 commits. Then, if I reloaded the index, it is as fast
as before! And today I have read a similar thread [1] and indeed: if I
set autowarming for the caches to 0 the slowdown disappears.
BUT at the same time I would like to offer searching on that core, which
would be dramatically slowed down (due to no autowarming).
Does someone know a better solution to avoid index-slow-down?
Regards,
Peter.
[1] http://www.mail-archive.com/solr-user@lucene.apache.org/msg20785.html
Re: Solr Indexing slows down
Posted by Peter Karich <pe...@yahoo.de>.
Thanks Otis, for this clarification!
That means I will have to descrease the commit frequency to speed up
indexing.
How could I do this if I don't want to introduce an artificial delay
time? ... via increasing the batch size?
Today I have read in another thread[1] that one should univert the
field? What is it and how can I do this?
Regards,
Peter.
[1]
http://www.mail-archive.com/solr-user@lucene.apache.org/msg36113.html
> As you make changes to your index, you probably want to see the new/modified
> documents in your search results. In order to do that, the new searcher needs
> to be reopened, and this happens on commit.
> Otis
> ----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Lucene ecosystem search :: http://search-lucene.com/
>
>
>
> ----- Original Message ----
>
>> From: Peter Karich <pe...@yahoo.de>
>> To: solr-user@lucene.apache.org
>> Sent: Fri, July 30, 2010 6:19:03 PM
>> Subject: Re: Solr Indexing slows down
>>
>> Hi Otis,
>>
>> does it mean that a new searcher is opened after I commit?
>> I thought only on startup...(?)
>>
>> Regards,
>> Peter.
>>
>>
>>> Peter, there are events in solrconfig where you define warm up queries when
>>>
>> a
>>
>>
>>> new searcher is opened.
>>>
>>> There are also cache settings that play a role here.
>>>
>>> 30-60 seconds is pretty frequent for Solr.
>>>
>>> Otis
>>> ----
>>> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
>>> Lucene ecosystem search :: http://search-lucene.com/
>>>
>>>
>>>
>>> ----- Original Message ----
>>>
>>>
>>>> From: Peter Karich <pe...@yahoo.de>
>>>> To: solr-user@lucene.apache.org
>>>> Sent: Fri, July 30, 2010 4:06:48 PM
>>>> Subject: Re: Solr Indexing slows down
>>>>
>>>> Hi Erick!
>>>>
>>>> thanks for the response!
>>>> I will answer your questions ;-)
>>>>
>>>>
>>>>
>>>>> How often are you making changes to your index?
>>>>>
>>>>>
>>>> Every 30-60 seconds. Too heavy?
>>>>
>>>>
>>>>
>>>>
>>>>> Do you have autocommit on?
>>>>>
>>>>>
>>>> No.
>>>>
>>>>
>>>>
>>>>
>>>>> Do you commit when updating each document?
>>>>>
>>>>>
>>>> No. I commit after a batch update of 200 documents
>>>>
>>>>
>>>>
>>>>
>>>>> Committing too often and consequently firing off warmup queries is the
>>>>>
>> first
>>
>>
>>>>>
>>>>>
>>>> place I'd look.
>>>>
>>>> Why is commiting firing warmup queries? Is there any documentation about
>>>> this subject?
>>>> How can I be sure that the previous commit has done its magic?
>>>>
>>>>
>>>>
>>>>> there are several config values that influence the commit frequency
>>>>>
>>>>>
>>>> I now know the autowarm and the mergeFactor config. What else? Is this
>>>> documentation complete:
>>>> http://wiki.apache.org/lucene-java/ImproveIndexingSpeed ?
>>>>
>>>> Regards,
>>>> Peter.
>>>>
>>>>
>>>>
>>>>> See the subject about 1500 threads. The first place I'd look is how
>>>>> often you're committing. If you're committing before the warmup queries
>>>>> from the previous commit have done their magic, you might be getting
>>>>> into a death spiral.
>>>>>
>>>>> HTH
>>>>> Erick
>>>>>
>>>>> On Thu, Jul 29, 2010 at 7:02 AM, Peter Karich <pe...@yahoo.de>
>>>>>
> wrote:
>
>>>>>
>>>>>
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am indexing a solr 1.4.0 core and commiting gets slower and slower.
>>>>>> Starting from 3-5 seconds for ~200 documents and ending with over 60
>>>>>> seconds after 800 commits. Then, if I reloaded the index, it is as
>>>>>>
> fast
>
>>>>>> as before! And today I have read a similar thread [1] and indeed: if I
>>>>>> set autowarming for the caches to 0 the slowdown disappears.
>>>>>>
>>>>>> BUT at the same time I would like to offer searching on that core,
>>>>>>
> which
>
>>>>>> would be dramatically slowed down (due to no autowarming).
>>>>>>
>>>>>> Does someone know a better solution to avoid index-slow-down?
>>>>>>
>>>>>> Regards,
>>>>>> Peter.
>>>>>>
>>>>>> [1] http://www.mail-archive.com/solr-user@lucene.apache.org/msg20785.html
>>>>>>
>>>>>>
>>>>>>
Re: Solr Indexing slows down
Posted by Otis Gospodnetic <ot...@yahoo.com>.
As you make changes to your index, you probably want to see the new/modified
documents in your search results. In order to do that, the new searcher needs
to be reopened, and this happens on commit.
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/
----- Original Message ----
> From: Peter Karich <pe...@yahoo.de>
> To: solr-user@lucene.apache.org
> Sent: Fri, July 30, 2010 6:19:03 PM
> Subject: Re: Solr Indexing slows down
>
> Hi Otis,
>
> does it mean that a new searcher is opened after I commit?
> I thought only on startup...(?)
>
> Regards,
> Peter.
>
> > Peter, there are events in solrconfig where you define warm up queries when
>a
>
> > new searcher is opened.
> >
> > There are also cache settings that play a role here.
> >
> > 30-60 seconds is pretty frequent for Solr.
> >
> > Otis
> > ----
> > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> > Lucene ecosystem search :: http://search-lucene.com/
> >
> >
> >
> > ----- Original Message ----
> >
> >> From: Peter Karich <pe...@yahoo.de>
> >> To: solr-user@lucene.apache.org
> >> Sent: Fri, July 30, 2010 4:06:48 PM
> >> Subject: Re: Solr Indexing slows down
> >>
> >> Hi Erick!
> >>
> >> thanks for the response!
> >> I will answer your questions ;-)
> >>
> >>
> >>> How often are you making changes to your index?
> >>>
> >> Every 30-60 seconds. Too heavy?
> >>
> >>
> >>
> >>> Do you have autocommit on?
> >>>
> >> No.
> >>
> >>
> >>
> >>> Do you commit when updating each document?
> >>>
> >> No. I commit after a batch update of 200 documents
> >>
> >>
> >>
> >>> Committing too often and consequently firing off warmup queries is the
>first
>
> >>>
> >> place I'd look.
> >>
> >> Why is commiting firing warmup queries? Is there any documentation about
> >> this subject?
> >> How can I be sure that the previous commit has done its magic?
> >>
> >>
> >>> there are several config values that influence the commit frequency
> >>>
> >>
> >> I now know the autowarm and the mergeFactor config. What else? Is this
> >> documentation complete:
> >> http://wiki.apache.org/lucene-java/ImproveIndexingSpeed ?
> >>
> >> Regards,
> >> Peter.
> >>
> >>
> >>> See the subject about 1500 threads. The first place I'd look is how
> >>> often you're committing. If you're committing before the warmup queries
> >>> from the previous commit have done their magic, you might be getting
> >>> into a death spiral.
> >>>
> >>> HTH
> >>> Erick
> >>>
> >>> On Thu, Jul 29, 2010 at 7:02 AM, Peter Karich <pe...@yahoo.de>
wrote:
> >>>
> >>>
> >>>
> >>>> Hi,
> >>>>
> >>>> I am indexing a solr 1.4.0 core and commiting gets slower and slower.
> >>>> Starting from 3-5 seconds for ~200 documents and ending with over 60
> >>>> seconds after 800 commits. Then, if I reloaded the index, it is as
fast
> >>>> as before! And today I have read a similar thread [1] and indeed: if I
> >>>> set autowarming for the caches to 0 the slowdown disappears.
> >>>>
> >>>> BUT at the same time I would like to offer searching on that core,
which
> >>>> would be dramatically slowed down (due to no autowarming).
> >>>>
> >>>> Does someone know a better solution to avoid index-slow-down?
> >>>>
> >>>> Regards,
> >>>> Peter.
> >>>>
> >>>> [1] http://www.mail-archive.com/solr-user@lucene.apache.org/msg20785.html
> >>>>
> >>>>
> >>>>
> >>
> >
>
>
> --
> http://karussell.wordpress.com/
>
>
Re: Solr Indexing slows down
Posted by Peter Karich <pe...@yahoo.de>.
Hi Otis,
does it mean that a new searcher is opened after I commit?
I thought only on startup...(?)
Regards,
Peter.
> Peter, there are events in solrconfig where you define warm up queries when a
> new searcher is opened.
>
> There are also cache settings that play a role here.
>
> 30-60 seconds is pretty frequent for Solr.
>
> Otis
> ----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Lucene ecosystem search :: http://search-lucene.com/
>
>
>
> ----- Original Message ----
>
>> From: Peter Karich <pe...@yahoo.de>
>> To: solr-user@lucene.apache.org
>> Sent: Fri, July 30, 2010 4:06:48 PM
>> Subject: Re: Solr Indexing slows down
>>
>> Hi Erick!
>>
>> thanks for the response!
>> I will answer your questions ;-)
>>
>>
>>> How often are you making changes to your index?
>>>
>> Every 30-60 seconds. Too heavy?
>>
>>
>>
>>> Do you have autocommit on?
>>>
>> No.
>>
>>
>>
>>> Do you commit when updating each document?
>>>
>> No. I commit after a batch update of 200 documents
>>
>>
>>
>>> Committing too often and consequently firing off warmup queries is the first
>>>
>> place I'd look.
>>
>> Why is commiting firing warmup queries? Is there any documentation about
>> this subject?
>> How can I be sure that the previous commit has done its magic?
>>
>>
>>> there are several config values that influence the commit frequency
>>>
>>
>> I now know the autowarm and the mergeFactor config. What else? Is this
>> documentation complete:
>> http://wiki.apache.org/lucene-java/ImproveIndexingSpeed ?
>>
>> Regards,
>> Peter.
>>
>>
>>> See the subject about 1500 threads. The first place I'd look is how
>>> often you're committing. If you're committing before the warmup queries
>>> from the previous commit have done their magic, you might be getting
>>> into a death spiral.
>>>
>>> HTH
>>> Erick
>>>
>>> On Thu, Jul 29, 2010 at 7:02 AM, Peter Karich <pe...@yahoo.de> wrote:
>>>
>>>
>>>
>>>> Hi,
>>>>
>>>> I am indexing a solr 1.4.0 core and commiting gets slower and slower.
>>>> Starting from 3-5 seconds for ~200 documents and ending with over 60
>>>> seconds after 800 commits. Then, if I reloaded the index, it is as fast
>>>> as before! And today I have read a similar thread [1] and indeed: if I
>>>> set autowarming for the caches to 0 the slowdown disappears.
>>>>
>>>> BUT at the same time I would like to offer searching on that core, which
>>>> would be dramatically slowed down (due to no autowarming).
>>>>
>>>> Does someone know a better solution to avoid index-slow-down?
>>>>
>>>> Regards,
>>>> Peter.
>>>>
>>>> [1] http://www.mail-archive.com/solr-user@lucene.apache.org/msg20785.html
>>>>
>>>>
>>>>
>>
>
--
http://karussell.wordpress.com/
Re: Solr Indexing slows down
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Peter, there are events in solrconfig where you define warm up queries when a
new searcher is opened.
There are also cache settings that play a role here.
30-60 seconds is pretty frequent for Solr.
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/
----- Original Message ----
> From: Peter Karich <pe...@yahoo.de>
> To: solr-user@lucene.apache.org
> Sent: Fri, July 30, 2010 4:06:48 PM
> Subject: Re: Solr Indexing slows down
>
> Hi Erick!
>
> thanks for the response!
> I will answer your questions ;-)
>
> > How often are you making changes to your index?
>
> Every 30-60 seconds. Too heavy?
>
>
> > Do you have autocommit on?
>
> No.
>
>
> > Do you commit when updating each document?
>
> No. I commit after a batch update of 200 documents
>
>
> > Committing too often and consequently firing off warmup queries is the first
>place I'd look.
>
> Why is commiting firing warmup queries? Is there any documentation about
> this subject?
> How can I be sure that the previous commit has done its magic?
>
> > there are several config values that influence the commit frequency
>
>
> I now know the autowarm and the mergeFactor config. What else? Is this
> documentation complete:
> http://wiki.apache.org/lucene-java/ImproveIndexingSpeed ?
>
> Regards,
> Peter.
>
> > See the subject about 1500 threads. The first place I'd look is how
> > often you're committing. If you're committing before the warmup queries
> > from the previous commit have done their magic, you might be getting
> > into a death spiral.
> >
> > HTH
> > Erick
> >
> > On Thu, Jul 29, 2010 at 7:02 AM, Peter Karich <pe...@yahoo.de> wrote:
> >
> >
> >> Hi,
> >>
> >> I am indexing a solr 1.4.0 core and commiting gets slower and slower.
> >> Starting from 3-5 seconds for ~200 documents and ending with over 60
> >> seconds after 800 commits. Then, if I reloaded the index, it is as fast
> >> as before! And today I have read a similar thread [1] and indeed: if I
> >> set autowarming for the caches to 0 the slowdown disappears.
> >>
> >> BUT at the same time I would like to offer searching on that core, which
> >> would be dramatically slowed down (due to no autowarming).
> >>
> >> Does someone know a better solution to avoid index-slow-down?
> >>
> >> Regards,
> >> Peter.
> >>
> >> [1] http://www.mail-archive.com/solr-user@lucene.apache.org/msg20785.html
> >>
> >>
>
Re: Solr Indexing slows down
Posted by Peter Karich <pe...@yahoo.de>.
Hi Erick!
thanks for the response!
I will answer your questions ;-)
> How often are you making changes to your index?
Every 30-60 seconds. Too heavy?
> Do you have autocommit on?
No.
> Do you commit when updating each document?
No. I commit after a batch update of 200 documents
> Committing too often and consequently firing off warmup queries is the first place I'd look.
Why is commiting firing warmup queries? Is there any documentation about
this subject?
How can I be sure that the previous commit has done its magic?
> there are several config values that influence the commit frequency
I now know the autowarm and the mergeFactor config. What else? Is this
documentation complete:
http://wiki.apache.org/lucene-java/ImproveIndexingSpeed ?
Regards,
Peter.
> See the subject about 1500 threads. The first place I'd look is how
> often you're committing. If you're committing before the warmup queries
> from the previous commit have done their magic, you might be getting
> into a death spiral.
>
> HTH
> Erick
>
> On Thu, Jul 29, 2010 at 7:02 AM, Peter Karich <pe...@yahoo.de> wrote:
>
>
>> Hi,
>>
>> I am indexing a solr 1.4.0 core and commiting gets slower and slower.
>> Starting from 3-5 seconds for ~200 documents and ending with over 60
>> seconds after 800 commits. Then, if I reloaded the index, it is as fast
>> as before! And today I have read a similar thread [1] and indeed: if I
>> set autowarming for the caches to 0 the slowdown disappears.
>>
>> BUT at the same time I would like to offer searching on that core, which
>> would be dramatically slowed down (due to no autowarming).
>>
>> Does someone know a better solution to avoid index-slow-down?
>>
>> Regards,
>> Peter.
>>
>> [1] http://www.mail-archive.com/solr-user@lucene.apache.org/msg20785.html
>>
>>
Re: Solr Indexing slows down
Posted by Erick Erickson <er...@gmail.com>.
See the subject about 1500 threads. The first place I'd look is how
often you're committing. If you're committing before the warmup queries
from the previous commit have done their magic, you might be getting
into a death spiral.
HTH
Erick
On Thu, Jul 29, 2010 at 7:02 AM, Peter Karich <pe...@yahoo.de> wrote:
> Hi,
>
> I am indexing a solr 1.4.0 core and commiting gets slower and slower.
> Starting from 3-5 seconds for ~200 documents and ending with over 60
> seconds after 800 commits. Then, if I reloaded the index, it is as fast
> as before! And today I have read a similar thread [1] and indeed: if I
> set autowarming for the caches to 0 the slowdown disappears.
>
> BUT at the same time I would like to offer searching on that core, which
> would be dramatically slowed down (due to no autowarming).
>
> Does someone know a better solution to avoid index-slow-down?
>
> Regards,
> Peter.
>
> [1] http://www.mail-archive.com/solr-user@lucene.apache.org/msg20785.html
>