Custom update processor and race condition with concurrent requests

[Sachin Divekar <sa...@merce.co.INVALID>]

Hi,

We are using Solr where there are many update operations. This may not be
the right use case for Solr but it's an old application and at this moment
we are in no mood to replace Solr with something else.

For one of our use case, we had to use optimistic concurrency for handling
concurrent updates. The implementation involves fetching a set of
documents, checking a field's value, then constructing the update
instruction with an appropriate version for optimistic concurrency, etc.

For reducing HTTP communication and network calls I have written a custom
update processor. We now push the request document blindly. The request
processor modifies the request document based on the documents already
present in the index e.g.

if field1 of indexedDoc == 'x'
  set field2 of reqDoc to 'y'
  index reqDoc

For testing, I'm processing 6k unique records (uniqueKey) with roughly 40k
updates on the same records in a random sequence. Ideally, at the end state
in Solr should be the same after every round of testing -- that's how the
algorithm is designed.

The client is written in Go. It works perfectly when I run less than four
goroutines (concurrent Solr requests). The end state of Solr is exactly the
same after every run. But, if I run more I see discrepancies with up to
four records out of 6k. The process finishes in 7 to 12 seconds. If I use
softCommit=true while pushing the updates to Solr the number of
discrepancies are fewer than

I think this the case of a race condition where when my update processor is
processing a request document it fetches the indexed or tlogged document,
another instance of the update processor due to concurrent request is
already modified the indexed document.

Does my understanding seem correct? If yes, is there any way this can be
achieved without using optimistic concurrency in the update processor?
Basically, can I somehow serialize the update requests in the update
processor?

That's a lot of information and also confusing, I guess. Please let me know
if any specific details or clarification is required.

Thank you.
Sachin

Re: Custom update processor and race condition with concurrent requests

[Sachin Divekar <ss...@gmail.com>]

So, the idea was that Solr serializes writes while appending records to
tlogs. Combining that with realtime gets to make decisions to modify the
request would achieve the result.

> I do wonder if it’s possible to insure that a given doc is always updated
from the same thread?  I’m assuming that the root of your issue is that
you’re pushing updates in parallel and the same doc is being updated from
two different places.

Yes, you are right. I am trying to find out a solution to this issue. But I
guess as Solr is not a transactional database this is not possible easily.
I was trying to do something by using the fact that Solr serializes writes
to tlogs.

On Wed, Mar 4, 2020 at 2:31 AM Erick Erickson <er...@gmail.com>
wrote:

> I guess I’m missing something. Assuming that S1 and S2 are sent in
> different batches from different threads from your client, there are any
> number of ways they could arrive out of order. Network delays, client
> delays, etc. So I don’t se any way to serialize them reliably.
>
> If they’re sent either in the same batch or by the same thread, then they
> should be sequential and I’d look again at your custom processor to see
> what’s happening there.
>
> I do wonder if it’s possible to insure that a given doc is always updated
> from the same thread? I’m assuming that the root of your issue is that
> you’re pushing updates in parallel and the same doc is being updated from
> two different places.
>
> Best,
> Erick
>
> > On Mar 3, 2020, at 11:23, Sachin Divekar <ss...@gmail.com> wrote:
> >
> > Thank, Erick.
> >
> > I think I was not clear enough. With the custom update processor, I'm not
> > using optimistic concurrency at all. The update processor just modifies
> the
> > incoming document with updated field values and atomic update
> instructions.
> > It then forwards the modified request further in the chain. So, just to
> be
> > clear in this test setup optimistic concurrency is not in the picture.
> >
> > However, it looks like if I want to run concurrent update requests I will
> > have to use optimistic concurrency, be it in update processor or in the
> > client. I was wondering if I can avoid that by serializing requests at
> the
> > update processor level.
> >
> >> Hmmm, _where_ is your custom update processor running? And is this
> > SolrCloud?
> > Currently, it's a single node Solr but eventually, it will be SolrCloud.
> I
> > am just testing the idea of doing something like this. Right now I am
> > running the custom update processor before DistributedProcessor in the
> > chain.
> >
> >> If you run it _after_ the update is distributed (i.e. insure it’ll run
> on
> > the leader) _and_ you can insure that your custom update processor is
> smart
> > enough to know which version of the document is the “right” one, I should
> > think you can get this to work.
> > I think that's the exact problem. My update processor fetches the
> document,
> > updates the request object and forwards it in the chain. The two
> concurrent
> > instances (S1 and S2) of the update processor can fetch the document, get
> > value 'x' of field 'f1' at the same time and process them whereas
> ideally,
> > S2 should see the value updated by S1.
> >
> > S1: fetches id1 -> gets f1: x -> sets f1: y -> Solr append it to tlog
> > S2: fetches id1 -> gets f1: x ...... ideally it should get 'y'
> >
> > Is that possible with UpdateProcessor? I am using realtimeget (
> > RealTimeGetComponent.getInputDocument()) in the update processor to fetch
> > the document.
> >
> >> You’ll have to use “real time get”, which fetches the most current
> > version of the document even if it hasn’t been committed and reject the
> > update if it’s too old. Anything in this path requires that the desired
> > update doesn’t depend on the value already having been changed by the
> first
> > update...
> >
> > In the case of multiple concurrent instances of the update processor
> > are RealTimeGetComponent.getInputDocument()
> > calls serialzed?
> >
> > thank you
> > Sachin
>

Re: Custom update processor and race condition with concurrent requests

[Erick Erickson <er...@gmail.com>]

I guess I’m missing something. Assuming that S1 and S2 are sent in different batches from different threads from your client, there are any number of ways they could arrive out of order. Network delays, client delays, etc. So I don’t se any way to serialize them reliably. 

If they’re sent either in the same batch or by the same thread, then they should be sequential and I’d look again at your custom processor to see what’s happening there.

I do wonder if it’s possible to insure that a given doc is always updated from the same thread? I’m assuming that the root of your issue is that you’re pushing updates in parallel and the same doc is being updated from two different places.

Best,
Erick

> On Mar 3, 2020, at 11:23, Sachin Divekar <ss...@gmail.com> wrote:
> 
> Thank, Erick.
> 
> I think I was not clear enough. With the custom update processor, I'm not
> using optimistic concurrency at all. The update processor just modifies the
> incoming document with updated field values and atomic update instructions.
> It then forwards the modified request further in the chain. So, just to be
> clear in this test setup optimistic concurrency is not in the picture.
> 
> However, it looks like if I want to run concurrent update requests I will
> have to use optimistic concurrency, be it in update processor or in the
> client. I was wondering if I can avoid that by serializing requests at the
> update processor level.
> 
>> Hmmm, _where_ is your custom update processor running? And is this
> SolrCloud?
> Currently, it's a single node Solr but eventually, it will be SolrCloud. I
> am just testing the idea of doing something like this. Right now I am
> running the custom update processor before DistributedProcessor in the
> chain.
> 
>> If you run it _after_ the update is distributed (i.e. insure it’ll run on
> the leader) _and_ you can insure that your custom update processor is smart
> enough to know which version of the document is the “right” one, I should
> think you can get this to work.
> I think that's the exact problem. My update processor fetches the document,
> updates the request object and forwards it in the chain. The two concurrent
> instances (S1 and S2) of the update processor can fetch the document, get
> value 'x' of field 'f1' at the same time and process them whereas ideally,
> S2 should see the value updated by S1.
> 
> S1: fetches id1 -> gets f1: x -> sets f1: y -> Solr append it to tlog
> S2: fetches id1 -> gets f1: x ...... ideally it should get 'y'
> 
> Is that possible with UpdateProcessor? I am using realtimeget (
> RealTimeGetComponent.getInputDocument()) in the update processor to fetch
> the document.
> 
>> You’ll have to use “real time get”, which fetches the most current
> version of the document even if it hasn’t been committed and reject the
> update if it’s too old. Anything in this path requires that the desired
> update doesn’t depend on the value already having been changed by the first
> update...
> 
> In the case of multiple concurrent instances of the update processor
> are RealTimeGetComponent.getInputDocument()
> calls serialzed?
> 
> thank you
> Sachin

Re: Custom update processor and race condition with concurrent requests

[Walter Underwood <wu...@wunderwood.org>]

This really, really looks like something that should be done with a
database, not with Solr. This assumes a transactional model, which
Solr doesn’t have. 

wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Mar 3, 2020, at 7:56 PM, Sachin Divekar <ss...@gmail.com> wrote:
> 
> Thanks for the reply, Chris. Sure, I will start from the beginning and
> explain the problem I'm trying to solve.
> 
> We have objects which we index in Solr. They go through state transitions
> based on various events in their life. But, the events can come out of
> sequence. So, to maintain that consistency we need to implement rules while
> updating the document state in Solr. e.g. if the old state is X and the new
> one is Y then update status field, if the old state is Y and the new one is
> X then do not update status field, etc.
> 
> This is a distributed system and the events of the same object can be
> produced on different nodes. They are updated into Solr on the same node.
> This is SolrCloud setup so these updates can be received by different Solr
> nodes.
> 
> We have already implemented it by using optimistic concurrency and realtime
> get. The client program runs on each node where the events are produced.
> Summary of the processing the client does as follows:
> 
> - the client batches multiple events
> - it uses _version_ to /update the records
> - based on various conflicts it modifies the records for which update failed
> - it /updates the modified records
> 
> That works fine but there is a lot of to and fro between the client and
> Solr and the implementation is complex.
> 
> So, I thought it can be simplified by moving this state transitions and
> processing logic into Solr by writing a custom update processor. The idea
> occurred to me when I was thinking about Solr serializing multiple
> concurrent requests for a document on the leader replica. So, my thought
> process was if I am getting this serialization for free I can implement the
> entire processing inside Solr and a dumb client to push records to Solr
> would be sufficient. But, that's not working. Perhaps the point I missed is
> that even though this processing is moved inside Solr I still have a race
> condition because of time-of-check to time-of-update gap.
> 
> While writing this it just occurred to me that I'm running my custom update
> processor before DistributedProcessor. I'm committing the same XY crime
> again but if I run it after DistributedProcessor can this race condition be
> avoided?
> 
> My secondary purpose in doing this exercise is to understand how Solr and
> distributed databases in general work. And that's the reason I am coming up
> with these hypotheses and try to validate them.
> 
> thanks
> Sachin
> 
> On Wed, Mar 4, 2020 at 12:09 AM Chris Hostetter <ho...@fucit.org>
> wrote:
> 
>> 
>> It sounds like fundementally the problem you have is that you want solr to
>> "block" all updates to docId=X ... at the update processor chain level ...
>> until an existing update is done.
>> 
>> but solr has no way to know that you want to block at that level.
>> 
>> ie: you asked...
>> 
>> : In the case of multiple concurrent instances of the update processor
>> : are RealTimeGetComponent.getInputDocument()
>> : calls serialzed?
>> 
>> ...but the answer to that question isn't really relevant, because
>> regardless of the answer, there is no garuntee at the java thread
>> scheduling level that the operations your custom code performs on the
>> results will happen in any particular order -- even if
>> RealTimeGetComponent.getInputDocument(42) where to block other concurrent
>> calls to RealTimeGetComponent.getInputDocument(42) that wouldn't ensure
>> that the custom code you have in Thread1 that calls that method will
>> finish it's modifications to the SolrInputDocument *before* the same
>> custom code in Thread2 calls RealTimeGetComponent.getInputDocument(42).
>> 
>> The only way to do something like this would be to add locking in your
>> custom code itself -- based on the uniqueKey of the document -- to say
>> "don't allow another thread to modify this document until i'm done" and
>> keep that lock held until the delegated processAdd call finishes (so you
>> know that the other update processors include RunUpdateProcessor has
>> finished) ... but that would only work (easily) in a single node
>> situation, in a multinode situation you'd have to first check the state of
>> the request and ensure that your processor (and it's locking logic) only
>> happen on the "leader" for that document, and deal with things at a
>> distributed level ... andyou've got a whole host of new headaches.
>> 
>> I would really suggest you take a step back and re-think your objectve,
>> and share with us the "end goal" you're trying to achieve with this custom
>> update processor, because it seems you may haveheaded down an
>> uneccessarily complex route.
>> 
>> what exactly is it you're trying to achieve?
>> 
>> https://people.apache.org/~hossman/#xyproblem
>> XY Problem
>> 
>> Your question appears to be an "XY Problem" ... that is: you are dealing
>> with "X", you are assuming "Y" will help you, and you are asking about "Y"
>> without giving more details about the "X" so that we can understand the
>> full issue.  Perhaps the best solution doesn't involve "Y" at all?
>> See Also: http://www.perlmonks.org/index.pl?node_id=542341
>> 
>> 
>> 
>> 
>> 
>> : Date: Tue, 3 Mar 2020 23:52:38 +0530
>> : From: Sachin Divekar <ss...@gmail.com>
>> : Reply-To: solr-user@lucene.apache.org
>> : To: solr-user@lucene.apache.org
>> : Subject: Re: Custom update processor and race condition with concurrent
>> :     requests
>> :
>> : Thank, Erick.
>> :
>> : I think I was not clear enough. With the custom update processor, I'm not
>> : using optimistic concurrency at all. The update processor just modifies
>> the
>> : incoming document with updated field values and atomic update
>> instructions.
>> : It then forwards the modified request further in the chain. So, just to
>> be
>> : clear in this test setup optimistic concurrency is not in the picture.
>> :
>> : However, it looks like if I want to run concurrent update requests I will
>> : have to use optimistic concurrency, be it in update processor or in the
>> : client. I was wondering if I can avoid that by serializing requests at
>> the
>> : update processor level.
>> :
>> : > Hmmm, _where_ is your custom update processor running? And is this
>> : SolrCloud?
>> : Currently, it's a single node Solr but eventually, it will be SolrCloud.
>> I
>> : am just testing the idea of doing something like this. Right now I am
>> : running the custom update processor before DistributedProcessor in the
>> : chain.
>> :
>> : > If you run it _after_ the update is distributed (i.e. insure it’ll run
>> on
>> : the leader) _and_ you can insure that your custom update processor is
>> smart
>> : enough to know which version of the document is the “right” one, I should
>> : think you can get this to work.
>> : I think that's the exact problem. My update processor fetches the
>> document,
>> : updates the request object and forwards it in the chain. The two
>> concurrent
>> : instances (S1 and S2) of the update processor can fetch the document, get
>> : value 'x' of field 'f1' at the same time and process them whereas
>> ideally,
>> : S2 should see the value updated by S1.
>> :
>> : S1: fetches id1 -> gets f1: x -> sets f1: y -> Solr append it to tlog
>> : S2: fetches id1 -> gets f1: x ...... ideally it should get 'y'
>> :
>> : Is that possible with UpdateProcessor? I am using realtimeget (
>> : RealTimeGetComponent.getInputDocument()) in the update processor to fetch
>> : the document.
>> :
>> : > You’ll have to use “real time get”, which fetches the most current
>> : version of the document even if it hasn’t been committed and reject the
>> : update if it’s too old. Anything in this path requires that the desired
>> : update doesn’t depend on the value already having been changed by the
>> first
>> : update...
>> :
>> : In the case of multiple concurrent instances of the update processor
>> : are RealTimeGetComponent.getInputDocument()
>> : calls serialzed?
>> :
>> : thank you
>> : Sachin
>> :
>> 
>> -Hoss
>> http://www.lucidworks.com/

Re: Custom update processor and race condition with concurrent requests

[Sachin Divekar <ss...@gmail.com>]

Thanks, Chris.

I think I should stop thinking about doing it in Solr. Anyway, I was just
trying to see how far I can go.

On Wed, Mar 4, 2020 at 11:50 PM Chris Hostetter <ho...@fucit.org>
wrote:

>
> : So, I thought it can be simplified by moving this state transitions and
> : processing logic into Solr by writing a custom update processor. The idea
> : occurred to me when I was thinking about Solr serializing multiple
> : concurrent requests for a document on the leader replica. So, my thought
> : process was if I am getting this serialization for free I can implement
> the
> : entire processing inside Solr and a dumb client to push records to Solr
> : would be sufficient. But, that's not working. Perhaps the point I missed
> is
> : that even though this processing is moved inside Solr I still have a race
> : condition because of time-of-check to time-of-update gap.
>
> Correct.  Solr is (hand wavy) "locking" updates to documents by id on the
> leader node to ensure they are transactional, but that locking happens
> inside DistributedUpdateProcessor, other update processors don't run
> "inside" that lock.
>

Understood. I was not thinking clearly about locking.


>
> : While writing this it just occurred to me that I'm running my custom
> update
> : processor before DistributedProcessor. I'm committing the same XY crime
> : again but if I run it after DistributedProcessor can this race condition
> be
> : avoided?
>
> no.  that would just introduce a whole new host of problems that are a
> much more ivolved conversation to get into (remeber: the processors after
> DUH run on every replica, after the leader has already assigned a
> version and said this update should go thorugh ... so now imagine what
> your error handling logic has to look like?)
>

I completely missed that post-processors run on every replica. It will be
too convoluted to implement.


>
>
> Ultimately the goal that you're talking about really feels like "business
> logic that requires syncronizing/blocking updates" but you're trying to
> avoid writing a syncronized client to do that syncronization and error
> handling before forwarding those updates to solr.
>
> I mean -- even with your explanation of your goal, there is a whole host
> of nuance / use case specific logic that has to go into "based on various
> conflicts it modifies the records for which update failed" -- and that
> logic seems like it would affect the locking: if you get a request that
> violates the legal state transition because of another request that
> (blocked it until it) just finished .... now what?  fail? apply some new
> rules?
>
> this seems like logic you should really want in a "middle ware" layer that
> your clients talk to and sends docs to solr.
>
> If you *REALLY* want to try and piggy back this logic into solr, then
> there is _one_ place i can think of where you can "hook in" to the logic
> DistributedUpdateHandler does while "locking" an id on the leader, and
> that would be extending the AtomicUpdateDocumentMerger...
>
> It's marked experimental, and I don't really understand the use cases
> for why it exists, and in order to customize this you would have to
> also subclass DistributedUpdateHandlerFactory to build your custom
> instance and pass it to the DUH constructor, but then -- in theory -- you
> could intercept any document update *after* the RTG, and before it's
> written to the TLOG, and apply some business logic.
>
> But i wouldn't recommend this ... "the'r be Dragons!"
>

Thanks for this explanation. Yes, that's too dangerous and really not worth
the effort.

I think I am concluding this exercise now. I will stick to my older
implementation where I am handling state transitions on the
clientside using optimistic locking.

--
Sachin

Re: Custom update processor and race condition with concurrent requests

[Chris Hostetter <ho...@fucit.org>]

: So, I thought it can be simplified by moving this state transitions and
: processing logic into Solr by writing a custom update processor. The idea
: occurred to me when I was thinking about Solr serializing multiple
: concurrent requests for a document on the leader replica. So, my thought
: process was if I am getting this serialization for free I can implement the
: entire processing inside Solr and a dumb client to push records to Solr
: would be sufficient. But, that's not working. Perhaps the point I missed is
: that even though this processing is moved inside Solr I still have a race
: condition because of time-of-check to time-of-update gap.

Correct.  Solr is (hand wavy) "locking" updates to documents by id on the 
leader node to ensure they are transactional, but that locking happens 
inside DistributedUpdateProcessor, other update processors don't run 
"inside" that lock.

: While writing this it just occurred to me that I'm running my custom update
: processor before DistributedProcessor. I'm committing the same XY crime
: again but if I run it after DistributedProcessor can this race condition be
: avoided?

no.  that would just introduce a whole new host of problems that are a 
much more ivolved conversation to get into (remeber: the processors after 
DUH run on every replica, after the leader has already assigned a 
version and said this update should go thorugh ... so now imagine what 
your error handling logic has to look like?)


Ultimately the goal that you're talking about really feels like "business 
logic that requires syncronizing/blocking updates" but you're trying to 
avoid writing a syncronized client to do that syncronization and error 
handling before forwarding those updates to solr.

I mean -- even with your explanation of your goal, there is a whole host 
of nuance / use case specific logic that has to go into "based on various 
conflicts it modifies the records for which update failed" -- and that 
logic seems like it would affect the locking: if you get a request that 
violates the legal state transition because of another request that 
(blocked it until it) just finished .... now what?  fail? apply some new 
rules?

this seems like logic you should really want in a "middle ware" layer that 
your clients talk to and sends docs to solr.

If you *REALLY* want to try and piggy back this logic into solr, then 
there is _one_ place i can think of where you can "hook in" to the logic 
DistributedUpdateHandler does while "locking" an id on the leader, and 
that would be extending the AtomicUpdateDocumentMerger...

It's marked experimental, and I don't really understand the use cases 
for why it exists, and in order to customize this you would have to 
also subclass DistributedUpdateHandlerFactory to build your custom 
instance and pass it to the DUH constructor, but then -- in theory -- you 
could intercept any document update *after* the RTG, and before it's 
written to the TLOG, and apply some business logic.

But i wouldn't recommend this ... "the'r be Dragons!"



-Hoss
http://www.lucidworks.com/

Re: Custom update processor and race condition with concurrent requests

[Sachin Divekar <ss...@gmail.com>]

Thanks for the reply, Chris. Sure, I will start from the beginning and
explain the problem I'm trying to solve.

We have objects which we index in Solr. They go through state transitions
based on various events in their life. But, the events can come out of
sequence. So, to maintain that consistency we need to implement rules while
updating the document state in Solr. e.g. if the old state is X and the new
one is Y then update status field, if the old state is Y and the new one is
X then do not update status field, etc.

This is a distributed system and the events of the same object can be
produced on different nodes. They are updated into Solr on the same node.
This is SolrCloud setup so these updates can be received by different Solr
nodes.

We have already implemented it by using optimistic concurrency and realtime
get. The client program runs on each node where the events are produced.
Summary of the processing the client does as follows:

- the client batches multiple events
- it uses _version_ to /update the records
- based on various conflicts it modifies the records for which update failed
- it /updates the modified records

That works fine but there is a lot of to and fro between the client and
Solr and the implementation is complex.

So, I thought it can be simplified by moving this state transitions and
processing logic into Solr by writing a custom update processor. The idea
occurred to me when I was thinking about Solr serializing multiple
concurrent requests for a document on the leader replica. So, my thought
process was if I am getting this serialization for free I can implement the
entire processing inside Solr and a dumb client to push records to Solr
would be sufficient. But, that's not working. Perhaps the point I missed is
that even though this processing is moved inside Solr I still have a race
condition because of time-of-check to time-of-update gap.

While writing this it just occurred to me that I'm running my custom update
processor before DistributedProcessor. I'm committing the same XY crime
again but if I run it after DistributedProcessor can this race condition be
avoided?

My secondary purpose in doing this exercise is to understand how Solr and
distributed databases in general work. And that's the reason I am coming up
with these hypotheses and try to validate them.

thanks
Sachin

On Wed, Mar 4, 2020 at 12:09 AM Chris Hostetter <ho...@fucit.org>
wrote:

>
> It sounds like fundementally the problem you have is that you want solr to
> "block" all updates to docId=X ... at the update processor chain level ...
> until an existing update is done.
>
> but solr has no way to know that you want to block at that level.
>
> ie: you asked...
>
> : In the case of multiple concurrent instances of the update processor
> : are RealTimeGetComponent.getInputDocument()
> : calls serialzed?
>
> ...but the answer to that question isn't really relevant, because
> regardless of the answer, there is no garuntee at the java thread
> scheduling level that the operations your custom code performs on the
> results will happen in any particular order -- even if
> RealTimeGetComponent.getInputDocument(42) where to block other concurrent
> calls to RealTimeGetComponent.getInputDocument(42) that wouldn't ensure
> that the custom code you have in Thread1 that calls that method will
> finish it's modifications to the SolrInputDocument *before* the same
> custom code in Thread2 calls RealTimeGetComponent.getInputDocument(42).
>
> The only way to do something like this would be to add locking in your
> custom code itself -- based on the uniqueKey of the document -- to say
> "don't allow another thread to modify this document until i'm done" and
> keep that lock held until the delegated processAdd call finishes (so you
> know that the other update processors include RunUpdateProcessor has
> finished) ... but that would only work (easily) in a single node
> situation, in a multinode situation you'd have to first check the state of
> the request and ensure that your processor (and it's locking logic) only
> happen on the "leader" for that document, and deal with things at a
> distributed level ... andyou've got a whole host of new headaches.
>
> I would really suggest you take a step back and re-think your objectve,
> and share with us the "end goal" you're trying to achieve with this custom
> update processor, because it seems you may haveheaded down an
> uneccessarily complex route.
>
> what exactly is it you're trying to achieve?
>
> https://people.apache.org/~hossman/#xyproblem
> XY Problem
>
> Your question appears to be an "XY Problem" ... that is: you are dealing
> with "X", you are assuming "Y" will help you, and you are asking about "Y"
> without giving more details about the "X" so that we can understand the
> full issue.  Perhaps the best solution doesn't involve "Y" at all?
> See Also: http://www.perlmonks.org/index.pl?node_id=542341
>
>
>
>
>
> : Date: Tue, 3 Mar 2020 23:52:38 +0530
> : From: Sachin Divekar <ss...@gmail.com>
> : Reply-To: solr-user@lucene.apache.org
> : To: solr-user@lucene.apache.org
> : Subject: Re: Custom update processor and race condition with concurrent
> :     requests
> :
> : Thank, Erick.
> :
> : I think I was not clear enough. With the custom update processor, I'm not
> : using optimistic concurrency at all. The update processor just modifies
> the
> : incoming document with updated field values and atomic update
> instructions.
> : It then forwards the modified request further in the chain. So, just to
> be
> : clear in this test setup optimistic concurrency is not in the picture.
> :
> : However, it looks like if I want to run concurrent update requests I will
> : have to use optimistic concurrency, be it in update processor or in the
> : client. I was wondering if I can avoid that by serializing requests at
> the
> : update processor level.
> :
> : > Hmmm, _where_ is your custom update processor running? And is this
> : SolrCloud?
> : Currently, it's a single node Solr but eventually, it will be SolrCloud.
> I
> : am just testing the idea of doing something like this. Right now I am
> : running the custom update processor before DistributedProcessor in the
> : chain.
> :
> : > If you run it _after_ the update is distributed (i.e. insure it’ll run
> on
> : the leader) _and_ you can insure that your custom update processor is
> smart
> : enough to know which version of the document is the “right” one, I should
> : think you can get this to work.
> : I think that's the exact problem. My update processor fetches the
> document,
> : updates the request object and forwards it in the chain. The two
> concurrent
> : instances (S1 and S2) of the update processor can fetch the document, get
> : value 'x' of field 'f1' at the same time and process them whereas
> ideally,
> : S2 should see the value updated by S1.
> :
> : S1: fetches id1 -> gets f1: x -> sets f1: y -> Solr append it to tlog
> : S2: fetches id1 -> gets f1: x ...... ideally it should get 'y'
> :
> : Is that possible with UpdateProcessor? I am using realtimeget (
> : RealTimeGetComponent.getInputDocument()) in the update processor to fetch
> : the document.
> :
> : > You’ll have to use “real time get”, which fetches the most current
> : version of the document even if it hasn’t been committed and reject the
> : update if it’s too old. Anything in this path requires that the desired
> : update doesn’t depend on the value already having been changed by the
> first
> : update...
> :
> : In the case of multiple concurrent instances of the update processor
> : are RealTimeGetComponent.getInputDocument()
> : calls serialzed?
> :
> : thank you
> : Sachin
> :
>
> -Hoss
> http://www.lucidworks.com/

Re: Custom update processor and race condition with concurrent requests

[Chris Hostetter <ho...@fucit.org>]

It sounds like fundementally the problem you have is that you want solr to 
"block" all updates to docId=X ... at the update processor chain level ... 
until an existing update is done.

but solr has no way to know that you want to block at that level.

ie: you asked...

: In the case of multiple concurrent instances of the update processor
: are RealTimeGetComponent.getInputDocument()
: calls serialzed?

...but the answer to that question isn't really relevant, because 
regardless of the answer, there is no garuntee at the java thread 
scheduling level that the operations your custom code performs on the 
results will happen in any particular order -- even if 
RealTimeGetComponent.getInputDocument(42) where to block other concurrent 
calls to RealTimeGetComponent.getInputDocument(42) that wouldn't ensure 
that the custom code you have in Thread1 that calls that method will 
finish it's modifications to the SolrInputDocument *before* the same 
custom code in Thread2 calls RealTimeGetComponent.getInputDocument(42).

The only way to do something like this would be to add locking in your 
custom code itself -- based on the uniqueKey of the document -- to say 
"don't allow another thread to modify this document until i'm done" and 
keep that lock held until the delegated processAdd call finishes (so you 
know that the other update processors include RunUpdateProcessor has 
finished) ... but that would only work (easily) in a single node 
situation, in a multinode situation you'd have to first check the state of 
the request and ensure that your processor (and it's locking logic) only 
happen on the "leader" for that document, and deal with things at a 
distributed level ... andyou've got a whole host of new headaches.

I would really suggest you take a step back and re-think your objectve, 
and share with us the "end goal" you're trying to achieve with this custom 
update processor, because it seems you may haveheaded down an 
uneccessarily complex route.  

what exactly is it you're trying to achieve?

https://people.apache.org/~hossman/#xyproblem
XY Problem

Your question appears to be an "XY Problem" ... that is: you are dealing
with "X", you are assuming "Y" will help you, and you are asking about "Y"
without giving more details about the "X" so that we can understand the
full issue.  Perhaps the best solution doesn't involve "Y" at all?
See Also: http://www.perlmonks.org/index.pl?node_id=542341





: Date: Tue, 3 Mar 2020 23:52:38 +0530
: From: Sachin Divekar <ss...@gmail.com>
: Reply-To: solr-user@lucene.apache.org
: To: solr-user@lucene.apache.org
: Subject: Re: Custom update processor and race condition with concurrent
:     requests
: 
: Thank, Erick.
: 
: I think I was not clear enough. With the custom update processor, I'm not
: using optimistic concurrency at all. The update processor just modifies the
: incoming document with updated field values and atomic update instructions.
: It then forwards the modified request further in the chain. So, just to be
: clear in this test setup optimistic concurrency is not in the picture.
: 
: However, it looks like if I want to run concurrent update requests I will
: have to use optimistic concurrency, be it in update processor or in the
: client. I was wondering if I can avoid that by serializing requests at the
: update processor level.
: 
: > Hmmm, _where_ is your custom update processor running? And is this
: SolrCloud?
: Currently, it's a single node Solr but eventually, it will be SolrCloud. I
: am just testing the idea of doing something like this. Right now I am
: running the custom update processor before DistributedProcessor in the
: chain.
: 
: > If you run it _after_ the update is distributed (i.e. insure it’ll run on
: the leader) _and_ you can insure that your custom update processor is smart
: enough to know which version of the document is the “right” one, I should
: think you can get this to work.
: I think that's the exact problem. My update processor fetches the document,
: updates the request object and forwards it in the chain. The two concurrent
: instances (S1 and S2) of the update processor can fetch the document, get
: value 'x' of field 'f1' at the same time and process them whereas ideally,
: S2 should see the value updated by S1.
: 
: S1: fetches id1 -> gets f1: x -> sets f1: y -> Solr append it to tlog
: S2: fetches id1 -> gets f1: x ...... ideally it should get 'y'
: 
: Is that possible with UpdateProcessor? I am using realtimeget (
: RealTimeGetComponent.getInputDocument()) in the update processor to fetch
: the document.
: 
: > You’ll have to use “real time get”, which fetches the most current
: version of the document even if it hasn’t been committed and reject the
: update if it’s too old. Anything in this path requires that the desired
: update doesn’t depend on the value already having been changed by the first
: update...
: 
: In the case of multiple concurrent instances of the update processor
: are RealTimeGetComponent.getInputDocument()
: calls serialzed?
: 
: thank you
: Sachin
: 

-Hoss
http://www.lucidworks.com/

Re: Custom update processor and race condition with concurrent requests

[Sachin Divekar <ss...@gmail.com>]

Thank, Erick.

I think I was not clear enough. With the custom update processor, I'm not
using optimistic concurrency at all. The update processor just modifies the
incoming document with updated field values and atomic update instructions.
It then forwards the modified request further in the chain. So, just to be
clear in this test setup optimistic concurrency is not in the picture.

However, it looks like if I want to run concurrent update requests I will
have to use optimistic concurrency, be it in update processor or in the
client. I was wondering if I can avoid that by serializing requests at the
update processor level.

> Hmmm, _where_ is your custom update processor running? And is this
SolrCloud?
Currently, it's a single node Solr but eventually, it will be SolrCloud. I
am just testing the idea of doing something like this. Right now I am
running the custom update processor before DistributedProcessor in the
chain.

> If you run it _after_ the update is distributed (i.e. insure it’ll run on
the leader) _and_ you can insure that your custom update processor is smart
enough to know which version of the document is the “right” one, I should
think you can get this to work.
I think that's the exact problem. My update processor fetches the document,
updates the request object and forwards it in the chain. The two concurrent
instances (S1 and S2) of the update processor can fetch the document, get
value 'x' of field 'f1' at the same time and process them whereas ideally,
S2 should see the value updated by S1.

S1: fetches id1 -> gets f1: x -> sets f1: y -> Solr append it to tlog
S2: fetches id1 -> gets f1: x ...... ideally it should get 'y'

Is that possible with UpdateProcessor? I am using realtimeget (
RealTimeGetComponent.getInputDocument()) in the update processor to fetch
the document.

> You’ll have to use “real time get”, which fetches the most current
version of the document even if it hasn’t been committed and reject the
update if it’s too old. Anything in this path requires that the desired
update doesn’t depend on the value already having been changed by the first
update...

In the case of multiple concurrent instances of the update processor
are RealTimeGetComponent.getInputDocument()
calls serialzed?

thank you
Sachin

Re: Custom update processor and race condition with concurrent requests

[Erick Erickson <er...@gmail.com>]

I don’t really see how this test setup can work, I think you’re just getting lucky with the 4 threads.

But let’s be specific about what optimistic concurrency is. If you update a document that has a _version_ field, and that document already exists with a value in the _version_ field higher than in the new doc, that doc will fail indexing.

Here’s the problem here, and I’m assuming that your test shell can get the same document in multiple threads and that’s where your problem is. If thread1 and thread2 have the same document with the same _version_ stamp, then one of the two updates will fail. But which one fails is entirely dependent on whether thread1 or thread2 sends the doc first, which Solr doesn’t control at all. If your custom update processor is changing the version stamp, all bets are off.

Hmmm, _where_ is your custom update processor running? And is this SolrCloud? If you run it _after_ the update is distributed (i.e. insure it’ll run on the leader) _and_ you can insure that your custom update processor is smart enough to know which version of the document is the “right” one, I should think you can get this to work. You’ll have to use “real time get”, which fetches the most current version of the document even if it hasn’t been committed and reject the update if it’s too old. Anything in this path requires that the desired update doesn’t depend on the value already having been changed by the first update...

Good luck,
Erick

> On Mar 3, 2020, at 09:54, Sachin Divekar <sa...@merce.co.invalid> wrote:
> 
> Hi,
> 
> We are using Solr where there are many update operations. This may not be
> the right use case for Solr but it's an old application and at this moment
> we are in no mood to replace Solr with something else.
> 
> For one of our use case, we had to use optimistic concurrency for handling
> concurrent updates. The implementation involves fetching a set of
> documents, checking a field's value, then constructing the update
> instruction with an appropriate version for optimistic concurrency, etc.
> 
> For reducing HTTP communication and network calls I have written a custom
> update processor. We now push the request document blindly. The request
> processor modifies the request document based on the documents already
> present in the index e.g.
> 
> if field1 of indexedDoc == 'x'
>  set field2 of reqDoc to 'y'
>  index reqDoc
> 
> For testing, I'm processing 6k unique records (uniqueKey) with roughly 40k
> updates on the same records in a random sequence. Ideally, at the end state
> in Solr should be the same after every round of testing -- that's how the
> algorithm is designed.
> 
> The client is written in Go. It works perfectly when I run less than four
> goroutines (concurrent Solr requests). The end state of Solr is exactly the
> same after every run. But, if I run more I see discrepancies with up to
> four records out of 6k. The process finishes in 7 to 12 seconds. If I use
> softCommit=true while pushing the updates to Solr the number of
> discrepancies are fewer than
> 
> I think this the case of a race condition where when my update processor is
> processing a request document it fetches the indexed or tlogged document,
> another instance of the update processor due to concurrent request is
> already modified the indexed document.
> 
> Does my understanding seem correct? If yes, is there any way this can be
> achieved without using optimistic concurrency in the update processor?
> Basically, can I somehow serialize the update requests in the update
> processor?
> 
> That's a lot of information and also confusing, I guess. Please let me know
> if any specific details or clarification is required.
> 
> Thank you.
> Sachin