Different update handlers for auto commit configuration

[danny teichthal <da...@gmail.com>]

Hi,
On our system we currently initiate a soft commit to SOLR after each
business transaction that initiate an update. Hard commits are automatic
each 2 minutes.
We want to limit the explicit commit and move to autoSoftCommit.

Because of business restrictions:
Online request should be available for searching after 2 seconds.
Update from batch jobs can be available after 10 seconds. (maybe more,
currently on discussion).
There are some transactions that must be available immediately.

Question
I thought about creating 3 different update handlers, each containing a
different autoSoftCommit configuration. Is this an acceptable solution, are
there any downsides in using multiple update handlers?

Thanks,

Re: Different update handlers for auto commit configuration

[danny teichthal <da...@gmail.com>]

Thanks for the clarification, I indeed mixed it with UpdateRequestHandler.

On Mon, Dec 1, 2014 at 11:24 PM, Chris Hostetter <ho...@fucit.org>
wrote:

>
> : I thought that the auto commit is per update handler because they are
> : configured within the update handler tag.
>
> <updateHandler> is not the same thing as a <requestHandler> that does
> updates.
>
> there can be many "Update request handlers" configured, but there is only
> ever one <updateHandler/> in a SolrCore.
>
>
>
> -Hoss
> http://www.lucidworks.com/
>

Re: Different update handlers for auto commit configuration

[Chris Hostetter <ho...@fucit.org>]

: I thought that the auto commit is per update handler because they are
: configured within the update handler tag.

<updateHandler> is not the same thing as a <requestHandler> that does 
updates.

there can be many "Update request handlers" configured, but there is only 
ever one <updateHandler/> in a SolrCore.



-Hoss
http://www.lucidworks.com/

Re: Different update handlers for auto commit configuration

[danny teichthal <da...@gmail.com>]

Thanks for the reply Erik,
I thought that the auto commit is per update handler because they are
configured within the update handler tag.
>From a little debugging  on SOLR code and it looked as if the Commit
Tracker will schedule a soft commit on update if it gets commitwithin
value, and if not it will use the autosofcommit timeUpperBound. So,
according to my understanding, the autosoftcommit should be equivalent to
the commit within.
Anyway, it looks like the commitWithin is more flexible, so I'll try it.

Regarding  the requirement, the problem is that we are suppose to be "Near
real time".
Since we are already in production it's harder to change the requirements.

You are right about the caches and warming, the cache is not being used
that much and we cannot do warming at all.
 But, the trigger for the change is that on load time the commit frequency
increases and we get the performance warnings.
 We may have a few commits in the same second.
So, as a first step if we set it to 2 we will have a benefit.
In addition, on our big cores we have an average of commit per second on 24
hours, so still there's some benefit.



On Mon, Dec 1, 2014 at 12:42 AM, Erick Erickson <er...@gmail.com>
wrote:

> Uhhhm, the soft/hard commit settings are global, not
> configured in each update handler.
>
> How are updates being done? Because if you're using SolrJ,
> you can just use the server.add(doclist, commitwithin) and it'll
> just be handled automatically.
>
> True, putting a 2 second commitwithin on an online update
> would pick up some batch updates that happened to come in, but
> that's probably OK.
>
> I'd also be sure that the 2 second requirement is real. Soft commits
> aren't as expensive as hard commits with openSearcher=true, but
> they aren't free either. At that fast a commit rate you probably won't
> get much benefit out of the top-level caches, and you'll be warming an
> awful lot.
>
> FWIW,
> Erick
>
> On Sun, Nov 30, 2014 at 12:32 PM, danny teichthal <da...@gmail.com>
> wrote:
> > Hi,
> > On our system we currently initiate a soft commit to SOLR after each
> > business transaction that initiate an update. Hard commits are automatic
> > each 2 minutes.
> > We want to limit the explicit commit and move to autoSoftCommit.
> >
> > Because of business restrictions:
> > Online request should be available for searching after 2 seconds.
> > Update from batch jobs can be available after 10 seconds. (maybe more,
> > currently on discussion).
> > There are some transactions that must be available immediately.
> >
> > Question
> > I thought about creating 3 different update handlers, each containing a
> > different autoSoftCommit configuration. Is this an acceptable solution,
> are
> > there any downsides in using multiple update handlers?
> >
> > Thanks,
>

Re: Different update handlers for auto commit configuration

[Erick Erickson <er...@gmail.com>]

Uhhhm, the soft/hard commit settings are global, not
configured in each update handler.

How are updates being done? Because if you're using SolrJ,
you can just use the server.add(doclist, commitwithin) and it'll
just be handled automatically.

True, putting a 2 second commitwithin on an online update
would pick up some batch updates that happened to come in, but
that's probably OK.

I'd also be sure that the 2 second requirement is real. Soft commits
aren't as expensive as hard commits with openSearcher=true, but
they aren't free either. At that fast a commit rate you probably won't
get much benefit out of the top-level caches, and you'll be warming an
awful lot.

FWIW,
Erick

On Sun, Nov 30, 2014 at 12:32 PM, danny teichthal <da...@gmail.com> wrote:
> Hi,
> On our system we currently initiate a soft commit to SOLR after each
> business transaction that initiate an update. Hard commits are automatic
> each 2 minutes.
> We want to limit the explicit commit and move to autoSoftCommit.
>
> Because of business restrictions:
> Online request should be available for searching after 2 seconds.
> Update from batch jobs can be available after 10 seconds. (maybe more,
> currently on discussion).
> There are some transactions that must be available immediately.
>
> Question
> I thought about creating 3 different update handlers, each containing a
> different autoSoftCommit configuration. Is this an acceptable solution, are
> there any downsides in using multiple update handlers?
>
> Thanks,