Commit behavior when softcommit is set at the server-level

[Christopher Schultz <ch...@christopherschultz.net>]

All,

We have our server set to soft-commit every 30 seconds.

If we know we are going to be sending a lot of documents at once, if we 
set "commitWithin" to something longer -- for those document-add/replace 
requests, will that override the soft-commit setting, or will the 
soft-commit setting just go-ahead and soft-commit regardless?

I'm assuming that the global soft-commit will simply happen since the 
documents are all going to the same place.

Given that we use Solr as an online index (meaning that we are 
constantly adding small numbers of documents to a comparatively large 
index), would it make more sense for us to:

1. Disable global soft-commit
2. Change our "document-add" operations to always specify commitWithin=30s

Doing steps 1-2 above seems like it would give us more control over 
processes where we know we are going to be adding lots of documents, and 
we can speed-up that indexing operation. It would also guarantee that we 
get a fairly up-to-date index to search while NOT committing after 
adding every single document (which I understand to be a horrible 
practice for performance).

We are currently using Solr 7.7, if that distinction is important. (For 
example, if commitWithin requires 8.x or later, or something like that.)

Thanks,
-chris

Re: Commit behavior when softcommit is set at the server-level

[Shawn Heisey <ap...@elyograg.org>]

On 6/16/22 09:16, Christopher Schultz wrote:
> We have our server set to soft-commit every 30 seconds.


30 seconds is very aggressive.  It's far better than 1 second, but still 
quite frequent.

> If we know we are going to be sending a lot of documents at once, if 
> we set "commitWithin" to something longer -- for those 
> document-add/replace requests, will that override the soft-commit 
> setting, or will the soft-commit setting just go-ahead and soft-commit 
> regardless?

All timers for committing are honored.  If you index some documents with 
autoSoftCommit at 30 seconds and commitWithin at 60 seconds, then the 
autoSoftCommit will fire 30 seconds after indexing begins, and 
commitWithin will fire 60 seconds after indexing begins.  If no changes 
take place between 30 and 60 seconds, then the commitWithin will be a no-op.

> 1. Disable global soft-commit
> 2. Change our "document-add" operations to always specify 
> commitWithin=30s

If the indexing requests ALWAYS have commitWithin (or even commit=true) 
then autoSoftCommit is superfluous.

I think what I would do is set autoSoftCommit at two to five minutes to 
catch any updates that manage to happen without a commit option, and 
make sure that all updates specify something for commit.  In that setup, 
autoSoftCommit will probably always be a no-op.

As I said ... 30 seconds is very aggressive, unless the index size is 
very small.  If the index has any heft, consider a larger interval.

FYI, commitWithin is typically executed as a soft commit.

Thanks,
Shawn