You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Shawn Heisey <ap...@elyograg.org.INVALID> on 2023/09/25 13:24:39 UTC
A general question about update ordering
I have a general question about update ordering.
Situation:
Two indexing requests for the same uniqueKey value sent. One update
"creates" the document, the other is an "update" with more fields
populated. The "update" is sent by a different system than the "create"
... and it turns out that they are sent in the wrong order, with the
update coming one millisecond BEFORE the create.
Solr 4.7 handles this as the customer wants, but Solr 9.1 doesn't.
I suspect that 4.7 is ignoring the order in which the updates were
received, so the indexing operation that takes longer is actually
applied after the one that happens faster ... while 9.1 is applying the
updates in the order they were received.
Is my idea correct? Is Lucene 9 or Solr 9 smarter than 4 about order of
operations?
Thanks,
Shawn
Re: A general question about update ordering
Posted by Shawn <el...@elyograg.org.INVALID>.
Verified. The _version_ field is not being sent in update requests.
On October 12, 2023 4:29:41 PM MDT, Shawn Heisey <el...@elyograg.org.INVALID> wrote:
>On 10/12/23 13:26, Chris Hostetter wrote:
>> i think you're missunderstanding rules of _version_ and how '0' is
>> handled.
>
>I'm about 90 percent sure the update requests are not being sent with the _version_ field populated. I am verifying this right as quickly as I can.
>
>Thanks,
>Shawn
>
Re: A general question about update ordering
Posted by Shawn Heisey <el...@elyograg.org.INVALID>.
On 10/12/23 13:26, Chris Hostetter wrote:
> i think you're missunderstanding rules of _version_ and how '0' is
> handled.
I'm about 90 percent sure the update requests are not being sent with
the _version_ field populated. I am verifying this right as quickly as
I can.
Thanks,
Shawn
Re: A general question about update ordering
Posted by Chris Hostetter <ho...@fucit.org>.
: Because the two requests are sent by different services and coordinating those
: requests woud likely be difficult, I think the simplest change that would fix
: the problem is to use optimistic concurrency, setting the field to 0 on the
: "create" request so the request fails if the document already exists. And of
i think you're missunderstanding rules of _version_ and how '0' is
handled.
_version_=0 just means "ignore any verion rules, add/overwrite this doc no
matter what"
what you probably want it...
* requests that expect to be creating a brand new doc: _version_=-1
"this doc must not already exist"
* requests that expect to be updating an existing doc: _version_=1
"this doc must exist with any version"
-Hoss
http://www.lucidworks.com/
Re: A general question about update ordering
Posted by Shawn Heisey <ap...@elyograg.org.INVALID>.
On 9/25/23 09:44, Walter Underwood wrote:
> Updates and indexing used to be single-threaded. A while ago (maybe in Solr 6.x?), a bunch of locking was removed so that indexing could use multiple CPUs. That is probably what you are seeing.
>
> I don’t think Solr has ever guaranteed ordering for updates, even if it happened to work.
>
> Maybe use a real-time get before the update to check whether the document is really there? That should be pretty fast.
Because the two requests are sent by different services and coordinating
those requests woud likely be difficult, I think the simplest change
that would fix the problem is to use optimistic concurrency, setting the
field to 0 on the "create" request so the request fails if the document
already exists. And of course the resulting error would need to be
explicitly detected so it can be ignored, while still correctly handling
other errors. I have not seen the SolrJ code that actually makes the
indexing requests.
In this, I am assuming, hopefully correctly, that optimistic concurrency
checks uncommitted docs like the /get handler, as it is very unlikely
that the earlier update has actually been committed when the second
update comes in.
Thanks,
Shawn
Re: A general question about update ordering
Posted by Walter Underwood <wu...@wunderwood.org>.
Updates and indexing used to be single-threaded. A while ago (maybe in Solr 6.x?), a bunch of locking was removed so that indexing could use multiple CPUs. That is probably what you are seeing.
I don’t think Solr has ever guaranteed ordering for updates, even if it happened to work.
Maybe use a real-time get before the update to check whether the document is really there? That should be pretty fast.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Sep 25, 2023, at 9:53 AM, Dmitri Maziuk <dm...@gmail.com> wrote:
>
> On 9/25/23 08:24, Shawn Heisey wrote:
> ...
>> Solr 4.7 handles this as the customer wants, but Solr 9.1 doesn't.
>
> This sounds like a race condition, are you sure it's Solr doing the ordering and not e.g. the network stack?
>
> What I found is that when the "add" request comes first, 8.11.2 (at least) will create a new -- useless -- document for it. I wish it failed instead.
>
> Dima
>
Re: A general question about update ordering
Posted by Dmitri Maziuk <dm...@gmail.com>.
On 9/25/23 11:35, Shawn Heisey wrote:
>
> I agree that the way they are operating now is completely wrong, and I
> am surprised that it even worked. They got VERY lucky.
http://www.catb.org/jargon/html/S/schroedinbug.html
cheers,
Dima
Re: A general question about update ordering
Posted by Shawn Heisey <el...@elyograg.org.INVALID>.
On 9/25/23 10:24, Dmitri Maziuk wrote:
> (From your description, relying on the 4.x behaviour was a bad idea in
> the first place: what if "create" request e.g. hit a bad sector on a
> non-TLER drive and was stuck on i/o for 100x longer?)
I agree that the way they are operating now is completely wrong, and I
am surprised that it even worked. They got VERY lucky.
Thanks,
Shawn
Re: A general question about update ordering
Posted by Dmitri Maziuk <dm...@gmail.com>.
On 9/25/23 10:49, Shawn Heisey wrote:
> On 9/25/23 08:53, Dmitri Maziuk wrote:
...
>> What I found is that when the "add" request comes first, 8.11.2 (at
>> least) will create a new -- useless -- document for it. I wish it
>> failed instead.
>
> You might be able to achieve the behavior you want with the optimistic
> concurrency feature:
Not in my case, but thanks for the tip: one more thing to file for
future reference.
Anyway, what Walter said: you're racing on threads.
(From your description, relying on the 4.x behaviour was a bad idea in
the first place: what if "create" request e.g. hit a bad sector on a
non-TLER drive and was stuck on i/o for 100x longer?)
Dima
Re: A general question about update ordering
Posted by Shawn Heisey <ap...@elyograg.org.INVALID>.
On 9/25/23 08:53, Dmitri Maziuk wrote:
> On 9/25/23 08:24, Shawn Heisey wrote:
> ...
>> Solr 4.7 handles this as the customer wants, but Solr 9.1 doesn't.
>
> This sounds like a race condition, are you sure it's Solr doing the
> ordering and not e.g. the network stack?
I have seen the requests in the logs on the two systems that send the
indexing requests and the matching entries in solr.log on 9.x. The
"update" request is sent one millisecond before the "create" request.
The "update" request has quite a bit more field data than the "create"
request, so I am sure it takes longer.
When these requests are sent to a 4.7 cluster, the end result is that
the updated record is kept. I believe this is happening due to how long
the two updates take, and the order of receipt is ignored.
When exactly the same thing is sent to a 9.1.1 cluster, the "create"
record is kept instead of the "update" record. The customer has said
that they think this is a bug in 9.1.1 ... but I think Solr 9 is
actually doing it right, honoring the order the updates were received,
and Solr 4 just happened to meet their expectations due to what I would
call a bug.
> What I found is that when the "add" request comes first, 8.11.2 (at
> least) will create a new -- useless -- document for it. I wish it failed
> instead.
You might be able to achieve the behavior you want with the optimistic
concurrency feature:
https://solr.apache.org/guide/solr/latest/indexing-guide/partial-document-updates.html#optimistic-concurrency
Maybe this feature can also help in my customer's setup.
Thanks,
Shawn
Re: A general question about update ordering
Posted by Dmitri Maziuk <dm...@gmail.com>.
On 9/25/23 08:24, Shawn Heisey wrote:
...
> Solr 4.7 handles this as the customer wants, but Solr 9.1 doesn't.
This sounds like a race condition, are you sure it's Solr doing the
ordering and not e.g. the network stack?
What I found is that when the "add" request comes first, 8.11.2 (at
least) will create a new -- useless -- document for it. I wish it failed
instead.
Dima