Best strategy to commit often under load.

[Jérôme Etévé <je...@gmail.com>]

Hi all,

 I've got a solr server under significant load ( ~40/s ) and a single
process which can potentially commit as often as possible.
Typically, when it commits every 5 or 10s, my solr server slows down
quite a lot and this can lead to congestion problems on my client
side.

What would you recommend in this situation, is it better to leave solr
performs the commits automatically with reasonable autocommit
parameters?

What are solr's best practices concerning this point?

Thanks for your help!

Jerome.

-- 
Jerome Eteve.
http://www.eteve.net
jerome@eteve.net

Re: Best strategy to commit often under load.

[Lance Norskog <go...@gmail.com>]

Yes, the Solr autoCommit feature is the right tool for this task. It
lets the indexing app be simple. In some earlier Solr releases, the
"number of documents" feature did not work, so you had to use the
"number of milliseconds" feature.

Note that each commit causes Solr to forget all cached data about the
index. So, you should turn off autowarming in the indexing instance.

Also, you don't need to commit very often. All data indexed is saved
before the commit. There are sites that index millions of records and
then do one commit.

On Tue, Sep 15, 2009 at 10:22 AM, Jason Rutherglen
<ja...@gmail.com> wrote:
> Hi Jerome,
>
> 5 seconds is too little using Solr 1.3 or 1.4 because of caching
> and segment warming. If you turn off caching and segment
> warming, then you may be able do 5s latency using either a
> RAMDirectory or an SSD. In the future these issues will be fixed
> and less than 1s will be possible.
>
> -J
>
> On Tue, Sep 15, 2009 at 3:07 AM, Jérôme Etévé <je...@gmail.com> wrote:
>> Hi all,
>>
>>  I've got a solr server under significant load ( ~40/s ) and a single
>> process which can potentially commit as often as possible.
>> Typically, when it commits every 5 or 10s, my solr server slows down
>> quite a lot and this can lead to congestion problems on my client
>> side.
>>
>> What would you recommend in this situation, is it better to leave solr
>> performs the commits automatically with reasonable autocommit
>> parameters?
>>
>> What are solr's best practices concerning this point?
>>
>> Thanks for your help!
>>
>> Jerome.
>>
>> --
>> Jerome Eteve.
>> http://www.eteve.net
>> jerome@eteve.net
>>
>



-- 
Lance Norskog
goksron@gmail.com

Re: Best strategy to commit often under load.

[Jason Rutherglen <ja...@gmail.com>]

Hi Jerome,

5 seconds is too little using Solr 1.3 or 1.4 because of caching
and segment warming. If you turn off caching and segment
warming, then you may be able do 5s latency using either a
RAMDirectory or an SSD. In the future these issues will be fixed
and less than 1s will be possible.

-J

On Tue, Sep 15, 2009 at 3:07 AM, Jérôme Etévé <je...@gmail.com> wrote:
> Hi all,
>
>  I've got a solr server under significant load ( ~40/s ) and a single
> process which can potentially commit as often as possible.
> Typically, when it commits every 5 or 10s, my solr server slows down
> quite a lot and this can lead to congestion problems on my client
> side.
>
> What would you recommend in this situation, is it better to leave solr
> performs the commits automatically with reasonable autocommit
> parameters?
>
> What are solr's best practices concerning this point?
>
> Thanks for your help!
>
> Jerome.
>
> --
> Jerome Eteve.
> http://www.eteve.net
> jerome@eteve.net
>