commitWithin UpdateProcessor for existing docs

[Dan Rosher <ro...@gmail.com>]

Hi,

We have a requirement to update existing docs live within say 30s.
New docs updated depending on solrconfig autoSoftCommit.

As our dev team are finding this difficult to implement within our
middleware, I was thinking of writing an update processor as a
'post-processor' to add commitWithin (unless already set) so it runs on
shard leader/replicas to add commitWithin as described above.

It seems like a reasonable idea to have existing updates visible within X
secs, wondering whether others have implemented something similar, or if
this is inherently a flawed idea?

Many thanks,
Dan

Re: commitWithin UpdateProcessor for existing docs

[Shawn Heisey <ap...@elyograg.org>]

On 12/14/21 3:46 AM, Dan Rosher wrote:
> We have a requirement to update existing docs live within say 30s.
> New docs updated depending on solrconfig autoSoftCommit.
>
> As our dev team are finding this difficult to implement within our
> middleware, I was thinking of writing an update processor as a
> 'post-processor' to add commitWithin (unless already set) so it runs on
> shard leader/replicas to add commitWithin as described above.
>
> It seems like a reasonable idea to have existing updates visible within X
> secs, wondering whether others have implemented something similar, or if
> this is inherently a flawed idea?

It would be better to rely on autoSoftCommit than writing and 
incorporating custom code.

Normally I would not suggest configuring autoSoftCommit with a maxTime 
so low, but if your index is small enough, and your server has enough 
memory, and you eliminate index/cache warming, you might be able to get 
commit speeds fast enough that a low autoSoftCommit interval will not 
cause problems.

Generally speaking, 30 seconds is much too aggressive a goal for change 
visibility.  But it might be achievable, depending on the use case.

Thanks,
Shawn