When to commit?

[UpAndGone <up...@web.de>]

Hi again,

 

Another question: Since a commit is expensive, it shouldn't be send after
each update, should it? Would it make sense to commit in a certain time
frame, let's say every 5 minutes? For a high traffic site, would five
minutes be okay or is it still to expensive?

 

Thanks

Christian

 

 

 

Re: When to commit?

[Bill Au <bi...@gmail.com>]

Also keep in mind that changes and additions to the index is not visible by
the
clients until a commit has occured.

Bill

On 6/28/06, Erik Hatcher <er...@ehatchersolutions.com> wrote:
>
>
> On Jun 28, 2006, at 3:43 PM, UpAndGone wrote:
> > Another question: Since a commit is expensive, it shouldn't be send
> > after
> > each update, should it? Would it make sense to commit in a certain
> > time
> > frame, let's say every 5 minutes? For a high traffic site, would five
> > minutes be okay or is it still to expensive?
>
> I do batch, pretty much one-shot, kinda indexing currently (loading
> about 40k documents currently) and I had to tune it to send a <commit/
> > frequently otherwise Solr would run out of memory.  I sent my
> indexer to send a commit every 100 documents.  The auto-warming is
> slightly expensive, but not currently prohibitively so, in my
> scenario but is rock solid.
>
> But, it's not really expensive from the outside clients perspective
> because Solr just keeps serving up things (very rapidly!) while auto-
> warming and then flips over to the updated index when warmed up.  So
> if you're doing intermittent incremental indexing you probably are
> fine just doing a commit for every change.  If you're doing batch
> indexing, spreading commits thinner is more appropriate.
>
> Just some off the cuff experiences, that's all.
>
>         Erik
>
>

Re: When to commit?

[Erik Hatcher <er...@ehatchersolutions.com>]

On Jun 28, 2006, at 3:43 PM, UpAndGone wrote:
> Another question: Since a commit is expensive, it shouldn't be send  
> after
> each update, should it? Would it make sense to commit in a certain  
> time
> frame, let's say every 5 minutes? For a high traffic site, would five
> minutes be okay or is it still to expensive?

I do batch, pretty much one-shot, kinda indexing currently (loading  
about 40k documents currently) and I had to tune it to send a <commit/ 
 > frequently otherwise Solr would run out of memory.  I sent my  
indexer to send a commit every 100 documents.  The auto-warming is  
slightly expensive, but not currently prohibitively so, in my  
scenario but is rock solid.

But, it's not really expensive from the outside clients perspective  
because Solr just keeps serving up things (very rapidly!) while auto- 
warming and then flips over to the updated index when warmed up.  So  
if you're doing intermittent incremental indexing you probably are  
fine just doing a commit for every change.  If you're doing batch  
indexing, spreading commits thinner is more appropriate.

Just some off the cuff experiences, that's all.

	Erik