Re: ConcurrentUpdate issue of solr 8.8.1

[Mark Miller <ma...@gmail.com>]

Given you say you saw a lot of this and that occurred after a specific
upgrade, I would guess your root issue is JavaBin.

Unfortunately, I don’t believe you can change internode communication to
another format if that’s the case.

But also if that is the case, it will be in the way of some work I have to
do over the coming months and so I will fix it.

Mark

On Wed, May 26, 2021 at 7:46 PM Mark Miller <ma...@gmail.com> wrote:

> You can run into this fairly easily even if you are doing everything right
> and none of your operations are intrinsically slow or problematic.
>
> DeleteByQuery, regardless of its cost, tends to be a large contributor,
> though not always. You can mitigate a bit with cautious, controlled use of
> it.
>
> I’m not surprised that http2 is even more prone to also being involved,
> though I didn’t think that client was yet using an http2 version, so that
> is a bit surprising, but these things can bleed over quite easily. Even
> more so in new Jetty versions.
>
> Jetty client -> server communication (and http2 in general) can be much
> more picky around handling connections that are not tightly managed for
> resuse under http2 (which can multiplex many requests over a single
> connection). If you don’t fully read input/output streams for example, the
> server doesn’t know that you don’t intend to finish dealing with your
> stream. It will wait some amount of time. And then it will whack that
> connection. There are all sorts of things that can manifest from this
> depending on all kinds of factors, but one of them is your client server
> communication can be hosed for a bit. Similarity things can happen even if
> you do always keep your connection pool connections in shape if say, you
> set a content length header that doesn’t match the content. You can do a
> lot poorly all day and hardly notice a peep unless you turn on debug
> logging for jetty or monitor tcp stats. And most of the time, things won’t
> be terrible as a result of it either. But every now and then you get pretty
> annoying consequences. And if you have something aggravating in the mix,
> maybe more than now and then.
>
> As I said though, you can run into this dist stall issue from a variety of
> ways, you can march down the list of what can cause it and low and behold,
> there that stall is again for another reason.
>
> I would try and tune towards what is working for you - http1 appears
> better - go with it. If you can use delete by query more sparingly, I would
> -perhaps batch them when updates are not so common, that’s a big
> instigator. Careful mixing in commits with it.
>
> You can check and make sure your server idle timeouts are higher than the
> clients - that can help a bit.
>
> If you are using the cloud client or even if you are not, you can hash on
> ids client side and send updates right to the leader they belong on to help
> reduce extraneous zig zag update traffic.
>
> In an intensive test world, this takes a large, multi pronged path to
> fully eliminate. In a production world under your control, you should be
> able to find a reasonable result with a bit of what you have been exploring
> and some acceptance for imperfection or use adjustment.
>
> Things can also change as Jetty dependencies are updated - though I will
> say in my experience not often for the better. A positive if one is working
> on development, perhaps less so in production. And again, even still, http
> 1.1 tends to be more forgiving. And delete by query and some other issues
> are still a bit more persistent, but less likely to move backwards or
> unexpectedly on you as the more ethereal connection issues and spurious EOF
> parsing exceptions.
>
> Many users are not heavily troubled by the above, so likely you will find
> a workable setup. That background is to essentially say, you may find clear
> sky’s tomorrow, but also, it’s not just some setup or use issue on your
> end, and also, while a single fix might set you up, keep in mind a single
> fix may be what your waiting for only by chance of your situation, so be
> proactive.
>
> MRM
>
> On Wed, May 19, 2021 at 8:48 AM Ding, Lehong (external - Project)
> <le...@sap.com.invalid> wrote:
>
>> *Background:*
>>
>> Before we moving to solr 8.8.1 from 7.7.2, we performed some performance
>> test on solr 8.8.1. We met a lot of concurrent update error in solr log.
>>
>> *Envrironment:*
>>
>> solrCloud with 3 cluster nodes with 500 collections, 5 collections have
>> about 10m documents.
>>
>> (1 shard, 3 replica)
>>
>> *Threads:*
>>
>> 30 update/add threads + 10 deleteByQuery threads
>>
>> *Results:*
>>
>> During deleteByQuery thread runing, only one node (lead node) has update
>> transactions, but other two node has none .
>>
>> *Errrors:  *
>>
>> java.io.IOException: Request processing has stalled for 20091ms with 100
>> remaining elements in the queue.java.io.IOException: Request processing has
>> stalled for 20091ms with 100 remaining elements in the queue. at
>> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:449)
>> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) at
>> org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:345)
>> at
>> org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:338)
>> at
>> org.apache.solr.update.SolrCmdDistributor.distribAdd(SolrCmdDistributor.java:244)
>> at
>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribAdd(DistributedZkUpdateProcessor.java:300)
>> at
>> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:230)
>> at
>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.processAdd(DistributedZkUpdateProcessor.java:245)
>> at
>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:106)
>> at
>> org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:110)
>> at
>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:343)
>> at
>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readIterator(JavaBinUpdateRequestCodec.java:291)
>> at
>> org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:338)
>> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:283)
>> at
>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readNamedList(JavaBinUpdateRequestCodec.java:244)
>>
>>
>>
>> * Temporary Solution:*
>>
>> adding -Dsolr.http1=1 in solr start parameters
>>
>> There are still some error in error log but the number is much small.
>>
>>
>>
>> *My Questions:*
>>
>> 1 We found solr cluster will eventually get the data consistent. What’s
>> the concurrent update error mainly impacted?
>>
>> 2 Adding  -Dsolr.http1=1 in solr start parameters can reduce the error
>> number. Do we realy need add this parameter? And does this parameter will
>> be kept in later version?
>>
>> Many Thanks.
>>
>>
>>
>>
>>
>> Thanks and Regards,
>>
>> Lehong Ding
>>
>>
>>
>>
>>
> --
> - Mark
>
> http://about.me/markrmiller
>
-- 
- Mark

http://about.me/markrmiller

Re: ConcurrentUpdate issue of solr 8.8.1

[Mark Miller <ma...@gmail.com>]

Fixing fastinputstream and javabin parsing may actually be as simple as
removing unsigned byte masks from the InputStream itself (some jetty fake
InputStream classes also unsign shift > 0 - the http client i InputStream
response handler for example - not sure if that would need to go) and
making sure all the masks happen at the site of bit shifts/logic. Then the
readFully always behavior could likely be removed and byte reads could be
readAndCheckByte that check for -1.

MRM

On Sat, Jun 12, 2021 at 2:01 PM Mark Miller <ma...@gmail.com> wrote:

> There are lots of actionable items here from my view.
>
> So many that I don’t see any general importance to any one of them on a
> large scale.
>
> I run into the same problem if someone where to say, let’s make sure all
> the tests are solid for the upcoming release. Well what does that mean?
> With little effort I can eek out tons of fails that are not usually
> prominent. I know tons of issues great and small that the tests almost
> never reveal that are worse than many of the fails. I know plenty of
> behavior and fails nix each other out. That poor performance and various
> behavior keeps some tests happy but doesn’t necessary translate to the real
> world.
>
> I can’t approach it as let’s make sure the tests are solid. But I can
> approach from directions like, let’s look at possible regressions in tests
> or behavior. Pain points in tests for a release, etc.
>
> For cross node communication, it’s the same. It’s not really an issue of
> “what are the problems”, but what are the problems that are currently in
> the way of what has to be supported. How much are they practically
> affecting the current expectations.
>
> My knowledges and approaches to many of these problems can be a bit
> difficult to translate. I’ve kind of ripped down javabin and streams and IO
> and built them back up. I use a mix of inputstreams that both stream and
> read/write data types and byte buffers and byte arrays via offheap buffers
> directly. JavaBin and transaction logs may use var int / zig zag encodings
> or direct byte buffer put/gets or intermix them. IO might happen
> sequentially like a stream or perhaps in parallel. The http client and
> jetty server do async and async IO and the off heap buffers for all of this
> or things like the bytes and reads/writes that back char sequences are all
> from my ByteBuffer pool. All to say, there are no direct translations for a
> lot of actionable items, nor is there a master list of what the practical
> affects and ramifications are in fairly different worlds.
>
> So in IMO, there are enough actionable items as to make most of them of
> relative low concern or priority to me. But if the list starts getting hit
> with many users having problems with connection resets and server stalls
> and javabin EOFException oddities or corrupt javabin outputs or something
> I’m supporting does,  I’m ready to take action.
>
> I don’t know how important or what to do about the strange FastInputStream
> and JavaBin. I have looked at other data input stream impls and they don’t
> behave this way. FastInputStreams also affect replication and transaction
> logs and various things.
>
> At the end of the day, you just have to make sure bit transformation, int
> byte to int auto casting doesn’t screw you. I have not found this to
> require the off contract readFully behavior or lack of -1 EOF checking that
> you don’t find elsewhere.
>
> I have ripped all of these pieces apart and gutted a lot of it.  I didn’t
> find fast InputStream or FastInputWriter and the like fast. I found often
> 3-4 or 5 layers of buffering depending on what was layered on what path.
> Each buffer allocating a byte array to be gc’d. 98% of the time writing and
> reading in memory or to *Jetty* ByteBuffer *buffers*, making the layered
> byte arrays being pumped out in between entirely not fast. And most of
> this, stream limited and oriented while sitting on tops of super fast and
> efficient NIO at all the end points. My reading is that they are called
> fast because mostly they remove synchronized methods - which have almost no
> cost uncontended anyway. So I take issue with the naming.
>
> Anyway,  I literally ripped a bunch of that chain out and reworked how
> most of the parts do IO - generally with no byte array gc pumping at all
> (the javabin utf conversations are also major offenders) and without any
> unnecessary layers or pretence of old style IO streams that don’t exist. If
> I need to stream something, I stream it, if I don’t, bye bye stream api,
> hello to the byte buffers that are actually there and all the bulk and
> direct off heap and concurrent access that allows for.
>
> And then http2 behavior and the client is the same thing. Had to rip and
> tear and rebuild and unbuild and expand and take over parts.
>
> So what’s actionable. Depending on what you want or need, 10000 things.
> But what’s necessary. I’m open to finding out based on the problems being
> hit, the likelihood and the frequency.
>
> MRM
>
>
>
> On Thu, Jun 10, 2021 at 8:58 AM Gus Heck <gu...@gmail.com> wrote:
>
>>
>> On Wed, Jun 9, 2021 at 12:25 AM Mark Miller <ma...@gmail.com>
>> wrote:
>>
>>>
>>> Instead of respecting stream contracts, JavaBin is given how much to
>>> read and it diligently works to read to that point or in some cases it’s
>>> own end marker.
>>>
>>>
>> That sounds actionable, should it be reading the available contents into
>> an array and letting javabin work with the array instead? Or perhaps to
>> avoid that, a javbin2 format needs a prefix/header section indicating
>> exactly how much needs to be read to keep http2 happy? I'd guess from your
>> comment that this currently gets into problems due to JVM version/os or
>> client/server version mismatches changing the length of something? Haven't
>> looked at javabin code in a long while so I'm mostly guessing...
>>
>> -Gus
>>
> --
> - Mark
>
> http://about.me/markrmiller
>
-- 
- Mark

http://about.me/markrmiller

Re: ConcurrentUpdate issue of solr 8.8.1

[Mark Miller <ma...@gmail.com>]

There are lots of actionable items here from my view.

So many that I don’t see any general importance to any one of them on a
large scale.

I run into the same problem if someone where to say, let’s make sure all
the tests are solid for the upcoming release. Well what does that mean?
With little effort I can eek out tons of fails that are not usually
prominent. I know tons of issues great and small that the tests almost
never reveal that are worse than many of the fails. I know plenty of
behavior and fails nix each other out. That poor performance and various
behavior keeps some tests happy but doesn’t necessary translate to the real
world.

I can’t approach it as let’s make sure the tests are solid. But I can
approach from directions like, let’s look at possible regressions in tests
or behavior. Pain points in tests for a release, etc.

For cross node communication, it’s the same. It’s not really an issue of
“what are the problems”, but what are the problems that are currently in
the way of what has to be supported. How much are they practically
affecting the current expectations.

My knowledges and approaches to many of these problems can be a bit
difficult to translate. I’ve kind of ripped down javabin and streams and IO
and built them back up. I use a mix of inputstreams that both stream and
read/write data types and byte buffers and byte arrays via offheap buffers
directly. JavaBin and transaction logs may use var int / zig zag encodings
or direct byte buffer put/gets or intermix them. IO might happen
sequentially like a stream or perhaps in parallel. The http client and
jetty server do async and async IO and the off heap buffers for all of this
or things like the bytes and reads/writes that back char sequences are all
from my ByteBuffer pool. All to say, there are no direct translations for a
lot of actionable items, nor is there a master list of what the practical
affects and ramifications are in fairly different worlds.

So in IMO, there are enough actionable items as to make most of them of
relative low concern or priority to me. But if the list starts getting hit
with many users having problems with connection resets and server stalls
and javabin EOFException oddities or corrupt javabin outputs or something
I’m supporting does,  I’m ready to take action.

I don’t know how important or what to do about the strange FastInputStream
and JavaBin. I have looked at other data input stream impls and they don’t
behave this way. FastInputStreams also affect replication and transaction
logs and various things.

At the end of the day, you just have to make sure bit transformation, int
byte to int auto casting doesn’t screw you. I have not found this to
require the off contract readFully behavior or lack of -1 EOF checking that
you don’t find elsewhere.

I have ripped all of these pieces apart and gutted a lot of it.  I didn’t
find fast InputStream or FastInputWriter and the like fast. I found often
3-4 or 5 layers of buffering depending on what was layered on what path.
Each buffer allocating a byte array to be gc’d. 98% of the time writing and
reading in memory or to *Jetty* ByteBuffer *buffers*, making the layered
byte arrays being pumped out in between entirely not fast. And most of
this, stream limited and oriented while sitting on tops of super fast and
efficient NIO at all the end points. My reading is that they are called
fast because mostly they remove synchronized methods - which have almost no
cost uncontended anyway. So I take issue with the naming.

Anyway,  I literally ripped a bunch of that chain out and reworked how most
of the parts do IO - generally with no byte array gc pumping at all (the
javabin utf conversations are also major offenders) and without any
unnecessary layers or pretence of old style IO streams that don’t exist. If
I need to stream something, I stream it, if I don’t, bye bye stream api,
hello to the byte buffers that are actually there and all the bulk and
direct off heap and concurrent access that allows for.

And then http2 behavior and the client is the same thing. Had to rip and
tear and rebuild and unbuild and expand and take over parts.

So what’s actionable. Depending on what you want or need, 10000 things. But
what’s necessary. I’m open to finding out based on the problems being hit,
the likelihood and the frequency.

MRM



On Thu, Jun 10, 2021 at 8:58 AM Gus Heck <gu...@gmail.com> wrote:

>
> On Wed, Jun 9, 2021 at 12:25 AM Mark Miller <ma...@gmail.com> wrote:
>
>>
>> Instead of respecting stream contracts, JavaBin is given how much to read
>> and it diligently works to read to that point or in some cases it’s own end
>> marker.
>>
>>
> That sounds actionable, should it be reading the available contents into
> an array and letting javabin work with the array instead? Or perhaps to
> avoid that, a javbin2 format needs a prefix/header section indicating
> exactly how much needs to be read to keep http2 happy? I'd guess from your
> comment that this currently gets into problems due to JVM version/os or
> client/server version mismatches changing the length of something? Haven't
> looked at javabin code in a long while so I'm mostly guessing...
>
> -Gus
>
-- 
- Mark

http://about.me/markrmiller

Re: ConcurrentUpdate issue of solr 8.8.1

[David Smiley <ds...@apache.org>]

FYI https://issues.apache.org/jira/browse/SOLR-15223 will deprecate Apache
HttpClient based SolrClients but I would hope that the Jetty HTTP Client
based SolrClients can be configured to use HTTP 1.1 if desired, although I
have not checked.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Thu, Jun 10, 2021 at 9:58 AM Gus Heck <gu...@gmail.com> wrote:

>
> On Wed, Jun 9, 2021 at 12:25 AM Mark Miller <ma...@gmail.com> wrote:
>
>>
>> Instead of respecting stream contracts, JavaBin is given how much to read
>> and it diligently works to read to that point or in some cases it’s own end
>> marker.
>>
>>
> That sounds actionable, should it be reading the available contents into
> an array and letting javabin work with the array instead? Or perhaps to
> avoid that, a javbin2 format needs a prefix/header section indicating
> exactly how much needs to be read to keep http2 happy? I'd guess from your
> comment that this currently gets into problems due to JVM version/os or
> client/server version mismatches changing the length of something? Haven't
> looked at javabin code in a long while so I'm mostly guessing...
>
> -Gus
>

Re: ConcurrentUpdate issue of solr 8.8.1

[Gus Heck <gu...@gmail.com>]

On Wed, Jun 9, 2021 at 12:25 AM Mark Miller <ma...@gmail.com> wrote:

>
> Instead of respecting stream contracts, JavaBin is given how much to read
> and it diligently works to read to that point or in some cases it’s own end
> marker.
>
>
That sounds actionable, should it be reading the available contents into an
array and letting javabin work with the array instead? Or perhaps to avoid
that, a javbin2 format needs a prefix/header section indicating exactly how
much needs to be read to keep http2 happy? I'd guess from your comment that
this currently gets into problems due to JVM version/os or client/server
version mismatches changing the length of something? Haven't looked at
javabin code in a long while so I'm mostly guessing...

-Gus

Re: ConcurrentUpdate issue of solr 8.8.1

[Mark Miller <ma...@gmail.com>]

Anyway, returning the original sender:

I had not notice dist updates *are* using the http2 concurrent update
client now.

So that makes a lot of sense on why http 1.1 force improves things. It’s
not nearly as sensitive to connection management.

So I I’d still recommend sticking to http1, but one thing that would
*perhaps* improve what you are seeing with http2 is to try configuring the
Jetty server to multiplex only a single request on a connection.

These issues can show worse behavior when multiple requests are multiplexed
over a single connection because when a connection has to be closed due to
connection management, all the requests multiplexing on that connection
take a hit. Http3 is supposed to solve this limitation.

Might be worth the experimentation just to see if it’s any better, but you
will still likely get similar behavior. If you file a Jira issue for this
though, the more you can try things and collect behavior results, the more
likely someone will be able to help out with a fix.

MRM

On Tue, Jun 8, 2021 at 11:25 PM Mark Miller <ma...@gmail.com> wrote:

> A few details on how JavaBin can be involved in this type of behavior:
>
> JavaBin consumes data by by wrapping InputStream’s with a FastInputStream.
> FastInputStream implements an InputStream, but it disregards the
> InputStream contract that says you have reached the end of the stream when
> -1 is returned. A -1 is just another bit of data to be shifted into a byte
> and either gum the works or corrupt the type or size read.
>
> Jetty does not give you a raw InputStream to the network that that will
> simply suck up data like a file from disk. You will get something different
> depending on the version, depending on the client, depending on http1 or 2
> in some cases, at least in terms of behavior.
>
> Generally you are going to get something backed by ByteBuffers pretending
> to be a raw stream.
>
> Depending on different circumstances, Jetty will decide to stop that
> stream by jumping in with a -1 when it hits circumstances that lead to
> killing the connection.
>
> The specifics for when and why will vary between http1 and 2. Http1 will
> often be using chunked encoding and have little idea about the the scope of
> the full content. It knows the size of each chunk. Http2 has no such
> encoding and works with sessions and frames and an entirely different
> protocol and needs to stop a channel or session. Multiplexing means this
> can affect things across requests in a way that is different than Http1. In
> Solr, you may also have different clients depending on the protocol with
> different retry implementations. Old Jetty clients offered retry, their
> view for the newer client is that it’s too application specific as an
> option, implement retry yourself if you want it. The Apache http1 client
> will retry based on the Solr retry configuration class.
>
> Given that JavaBin has little respect for streams and deals in expected
> sizes or it’s own end markers, there is plenty of room to not clear a
> request. Given we try and not close streams due to old webapp days of
> tomcat and earlier jetty versions, there is more room for poor behavior.
> Not closing streams is no longer helpful at all in most cases. Generally
> there is an attempt to read to end of stream, but this is a hit ir miss
> strategy, especially as currently done.
>
> Instead of respecting stream contracts, JavaBin is given how much to read
> and it diligently works to read to that point or in some cases it’s own end
> marker.
>
> So when Jetty can’t tell a stream is actually complete, it needs to
> timeout before it will forcibly kill it. Varying behavior between http 1
> and 2. For various reasons it may try to do this as for other reasons as
> well. That will cause negative affects (delays in many cases) and reset or
> closed exceptions. The -1 it will inject into the stream won’t generally
> get a prompt response either. Depending on timing and data, JavaBin may not
> even see this an early end and simply spit out something off.  Retries from
> the same client won’t help for some time in some cases (HTTP2 worse than 1)
> either. Mileage will vary depending on HTTP2 or 1.1 for most of this. Http2
> can send things like GoAway or session lifecycle info as well that will
> lead to poorly handled exceptions in a similar vein.
>
> There are other things that could be at play, but this is one that behaves
> worse with http2 and also appears worse with Jetty 10 / 11 than 9. The
> better the system acts, the more visible these problems tend to be.
>
> MRM
>
> On Fri, Jun 4, 2021 at 3:55 PM Mark Miller <ma...@gmail.com> wrote:
>
>> Given you say you saw a lot of this and that occurred after a specific
>> upgrade, I would guess your root issue is JavaBin.
>>
>> Unfortunately, I don’t believe you can change internode communication to
>> another format if that’s the case.
>>
>> But also if that is the case, it will be in the way of some work I have
>> to do over the coming months and so I will fix it.
>>
>> Mark
>>
>> On Wed, May 26, 2021 at 7:46 PM Mark Miller <ma...@gmail.com>
>> wrote:
>>
>>> You can run into this fairly easily even if you are doing everything
>>> right and none of your operations are intrinsically slow or problematic.
>>>
>>> DeleteByQuery, regardless of its cost, tends to be a large contributor,
>>> though not always. You can mitigate a bit with cautious, controlled use of
>>> it.
>>>
>>> I’m not surprised that http2 is even more prone to also being involved,
>>> though I didn’t think that client was yet using an http2 version, so that
>>> is a bit surprising, but these things can bleed over quite easily. Even
>>> more so in new Jetty versions.
>>>
>>> Jetty client -> server communication (and http2 in general) can be much
>>> more picky around handling connections that are not tightly managed for
>>> resuse under http2 (which can multiplex many requests over a single
>>> connection). If you don’t fully read input/output streams for example, the
>>> server doesn’t know that you don’t intend to finish dealing with your
>>> stream. It will wait some amount of time. And then it will whack that
>>> connection. There are all sorts of things that can manifest from this
>>> depending on all kinds of factors, but one of them is your client server
>>> communication can be hosed for a bit. Similarity things can happen even if
>>> you do always keep your connection pool connections in shape if say, you
>>> set a content length header that doesn’t match the content. You can do a
>>> lot poorly all day and hardly notice a peep unless you turn on debug
>>> logging for jetty or monitor tcp stats. And most of the time, things won’t
>>> be terrible as a result of it either. But every now and then you get pretty
>>> annoying consequences. And if you have something aggravating in the mix,
>>> maybe more than now and then.
>>>
>>> As I said though, you can run into this dist stall issue from a variety
>>> of ways, you can march down the list of what can cause it and low and
>>> behold, there that stall is again for another reason.
>>>
>>> I would try and tune towards what is working for you - http1 appears
>>> better - go with it. If you can use delete by query more sparingly, I would
>>> -perhaps batch them when updates are not so common, that’s a big
>>> instigator. Careful mixing in commits with it.
>>>
>>> You can check and make sure your server idle timeouts are higher than
>>> the clients - that can help a bit.
>>>
>>> If you are using the cloud client or even if you are not, you can hash
>>> on ids client side and send updates right to the leader they belong on to
>>> help reduce extraneous zig zag update traffic.
>>>
>>> In an intensive test world, this takes a large, multi pronged path to
>>> fully eliminate. In a production world under your control, you should be
>>> able to find a reasonable result with a bit of what you have been exploring
>>> and some acceptance for imperfection or use adjustment.
>>>
>>> Things can also change as Jetty dependencies are updated - though I will
>>> say in my experience not often for the better. A positive if one is working
>>> on development, perhaps less so in production. And again, even still, http
>>> 1.1 tends to be more forgiving. And delete by query and some other issues
>>> are still a bit more persistent, but less likely to move backwards or
>>> unexpectedly on you as the more ethereal connection issues and spurious EOF
>>> parsing exceptions.
>>>
>>> Many users are not heavily troubled by the above, so likely you will
>>> find a workable setup. That background is to essentially say, you may find
>>> clear sky’s tomorrow, but also, it’s not just some setup or use issue on
>>> your end, and also, while a single fix might set you up, keep in mind a
>>> single fix may be what your waiting for only by chance of your situation,
>>> so be proactive.
>>>
>>> MRM
>>>
>>> On Wed, May 19, 2021 at 8:48 AM Ding, Lehong (external - Project)
>>> <le...@sap.com.invalid> wrote:
>>>
>>>> *Background:*
>>>>
>>>> Before we moving to solr 8.8.1 from 7.7.2, we performed some
>>>> performance test on solr 8.8.1. We met a lot of concurrent update error in
>>>> solr log.
>>>>
>>>> *Envrironment:*
>>>>
>>>> solrCloud with 3 cluster nodes with 500 collections, 5 collections have
>>>> about 10m documents.
>>>>
>>>> (1 shard, 3 replica)
>>>>
>>>> *Threads:*
>>>>
>>>> 30 update/add threads + 10 deleteByQuery threads
>>>>
>>>> *Results:*
>>>>
>>>> During deleteByQuery thread runing, only one node (lead node) has
>>>> update transactions, but other two node has none .
>>>>
>>>> *Errrors:  *
>>>>
>>>> java.io.IOException: Request processing has stalled for 20091ms with
>>>> 100 remaining elements in the queue.java.io.IOException: Request processing
>>>> has stalled for 20091ms with 100 remaining elements in the queue. at
>>>> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:449)
>>>> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) at
>>>> org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:345)
>>>> at
>>>> org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:338)
>>>> at
>>>> org.apache.solr.update.SolrCmdDistributor.distribAdd(SolrCmdDistributor.java:244)
>>>> at
>>>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribAdd(DistributedZkUpdateProcessor.java:300)
>>>> at
>>>> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:230)
>>>> at
>>>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.processAdd(DistributedZkUpdateProcessor.java:245)
>>>> at
>>>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:106)
>>>> at
>>>> org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:110)
>>>> at
>>>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:343)
>>>> at
>>>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readIterator(JavaBinUpdateRequestCodec.java:291)
>>>> at
>>>> org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:338)
>>>> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:283)
>>>> at
>>>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readNamedList(JavaBinUpdateRequestCodec.java:244)
>>>>
>>>>
>>>>
>>>> * Temporary Solution:*
>>>>
>>>> adding -Dsolr.http1=1 in solr start parameters
>>>>
>>>> There are still some error in error log but the number is much small.
>>>>
>>>>
>>>>
>>>> *My Questions:*
>>>>
>>>> 1 We found solr cluster will eventually get the data consistent. What’s
>>>> the concurrent update error mainly impacted?
>>>>
>>>> 2 Adding  -Dsolr.http1=1 in solr start parameters can reduce the error
>>>> number. Do we realy need add this parameter? And does this parameter will
>>>> be kept in later version?
>>>>
>>>> Many Thanks.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Thanks and Regards,
>>>>
>>>> Lehong Ding
>>>>
>>>>
>>>>
>>>>
>>>>
>>> --
>>> - Mark
>>>
>>> http://about.me/markrmiller
>>>
>> --
>> - Mark
>>
>> http://about.me/markrmiller
>>
> --
> - Mark
>
> http://about.me/markrmiller
>
-- 
- Mark

http://about.me/markrmiller

Re: ConcurrentUpdate issue of solr 8.8.1

[Mark Miller <ma...@gmail.com>]

Though I should say: while I won’t file some large general jira issue about
all the problems that can more visibly manifest in connection fails and
stalls when moving from the Apache client to the jetty client, from the old
SolrClient impl to the Http2 impl, from Http1.1 protocol to 2, how JavaBin
can interact with that - I don’t even know how prominently they show up in
the current code base on a wide practical basis, but if they are bad enough
to show up in what I have to support, I’ll have to make fixes at some
point. Then you will see a JIRA issue.

On Thu, Jun 10, 2021 at 7:24 AM Mark Miller <ma...@gmail.com> wrote:

> There is no line of code to point to.
>
> I see in the stack trace now that SolrCmdDistributor was updated to the
> http2 client.
>
> Lots more stalls and exceptions seen unless that client is changed to use
> http1 instead of 2.
>
> But still a lesser problem remains with 1.1.
>
> That is a bit of info on why this would be possible by switching from the
> Apache client to the Jetty client. And why Http2 can be worse than one.
>
> Poor connection management. JavaBin additional aggravation.
>
> That’s not a line of code. Probably not even a single Jira issue.
>
> MRM
>
> On Wed, Jun 9, 2021 at 8:47 AM David Smiley <ds...@apache.org> wrote:
>
>> Mark, can you file a JIRA issue (if one doesn't already exist), ideally
>> pointing to a line of code that is problematic?
>>
>> Do you think we should drop the "closeShield" stuff in
>> SolrDispatchFilter?  Not sure if there is a test exercising the scenario
>> that led to its inclusion; I doubt it.
>>
>>
>> ~ David Smiley
>> Apache Lucene/Solr Search Developer
>> http://www.linkedin.com/in/davidwsmiley
>>
>>
>> On Wed, Jun 9, 2021 at 12:25 AM Mark Miller <ma...@gmail.com>
>> wrote:
>>
>>> A few details on how JavaBin can be involved in this type of behavior:
>>>
>>> JavaBin consumes data by by wrapping InputStream’s with a
>>> FastInputStream. FastInputStream implements an InputStream, but it
>>> disregards the InputStream contract that says you have reached the end of
>>> the stream when -1 is returned. A -1 is just another bit of data to be
>>> shifted into a byte and either gum the works or corrupt the type or size
>>> read.
>>>
>>> Jetty does not give you a raw InputStream to the network that that will
>>> simply suck up data like a file from disk. You will get something different
>>> depending on the version, depending on the client, depending on http1 or 2
>>> in some cases, at least in terms of behavior.
>>>
>>> Generally you are going to get something backed by ByteBuffers
>>> pretending to be a raw stream.
>>>
>>> Depending on different circumstances, Jetty will decide to stop that
>>> stream by jumping in with a -1 when it hits circumstances that lead to
>>> killing the connection.
>>>
>>> The specifics for when and why will vary between http1 and 2. Http1 will
>>> often be using chunked encoding and have little idea about the the scope of
>>> the full content. It knows the size of each chunk. Http2 has no such
>>> encoding and works with sessions and frames and an entirely different
>>> protocol and needs to stop a channel or session. Multiplexing means this
>>> can affect things across requests in a way that is different than Http1. In
>>> Solr, you may also have different clients depending on the protocol with
>>> different retry implementations. Old Jetty clients offered retry, their
>>> view for the newer client is that it’s too application specific as an
>>> option, implement retry yourself if you want it. The Apache http1 client
>>> will retry based on the Solr retry configuration class.
>>>
>>> Given that JavaBin has little respect for streams and deals in expected
>>> sizes or it’s own end markers, there is plenty of room to not clear a
>>> request. Given we try and not close streams due to old webapp days of
>>> tomcat and earlier jetty versions, there is more room for poor behavior.
>>> Not closing streams is no longer helpful at all in most cases. Generally
>>> there is an attempt to read to end of stream, but this is a hit ir miss
>>> strategy, especially as currently done.
>>>
>>> Instead of respecting stream contracts, JavaBin is given how much to
>>> read and it diligently works to read to that point or in some cases it’s
>>> own end marker.
>>>
>>> So when Jetty can’t tell a stream is actually complete, it needs to
>>> timeout before it will forcibly kill it. Varying behavior between http 1
>>> and 2. For various reasons it may try to do this as for other reasons as
>>> well. That will cause negative affects (delays in many cases) and reset or
>>> closed exceptions. The -1 it will inject into the stream won’t generally
>>> get a prompt response either. Depending on timing and data, JavaBin may not
>>> even see this an early end and simply spit out something off.  Retries from
>>> the same client won’t help for some time in some cases (HTTP2 worse than 1)
>>> either. Mileage will vary depending on HTTP2 or 1.1 for most of this. Http2
>>> can send things like GoAway or session lifecycle info as well that will
>>> lead to poorly handled exceptions in a similar vein.
>>>
>>> There are other things that could be at play, but this is one that
>>> behaves worse with http2 and also appears worse with Jetty 10 / 11 than 9.
>>> The better the system acts, the more visible these problems tend to be.
>>>
>>> MRM
>>>
>>> On Fri, Jun 4, 2021 at 3:55 PM Mark Miller <ma...@gmail.com>
>>> wrote:
>>>
>>>> Given you say you saw a lot of this and that occurred after a specific
>>>> upgrade, I would guess your root issue is JavaBin.
>>>>
>>>> Unfortunately, I don’t believe you can change internode communication
>>>> to another format if that’s the case.
>>>>
>>>> But also if that is the case, it will be in the way of some work I have
>>>> to do over the coming months and so I will fix it.
>>>>
>>>> Mark
>>>>
>>>> On Wed, May 26, 2021 at 7:46 PM Mark Miller <ma...@gmail.com>
>>>> wrote:
>>>>
>>>>> You can run into this fairly easily even if you are doing everything
>>>>> right and none of your operations are intrinsically slow or problematic.
>>>>>
>>>>> DeleteByQuery, regardless of its cost, tends to be a large
>>>>> contributor, though not always. You can mitigate a bit with cautious,
>>>>> controlled use of it.
>>>>>
>>>>> I’m not surprised that http2 is even more prone to also being
>>>>> involved, though I didn’t think that client was yet using an http2 version,
>>>>> so that is a bit surprising, but these things can bleed over quite easily.
>>>>> Even more so in new Jetty versions.
>>>>>
>>>>> Jetty client -> server communication (and http2 in general) can be
>>>>> much more picky around handling connections that are not tightly managed
>>>>> for resuse under http2 (which can multiplex many requests over a single
>>>>> connection). If you don’t fully read input/output streams for example, the
>>>>> server doesn’t know that you don’t intend to finish dealing with your
>>>>> stream. It will wait some amount of time. And then it will whack that
>>>>> connection. There are all sorts of things that can manifest from this
>>>>> depending on all kinds of factors, but one of them is your client server
>>>>> communication can be hosed for a bit. Similarity things can happen even if
>>>>> you do always keep your connection pool connections in shape if say, you
>>>>> set a content length header that doesn’t match the content. You can do a
>>>>> lot poorly all day and hardly notice a peep unless you turn on debug
>>>>> logging for jetty or monitor tcp stats. And most of the time, things won’t
>>>>> be terrible as a result of it either. But every now and then you get pretty
>>>>> annoying consequences. And if you have something aggravating in the mix,
>>>>> maybe more than now and then.
>>>>>
>>>>> As I said though, you can run into this dist stall issue from a
>>>>> variety of ways, you can march down the list of what can cause it and low
>>>>> and behold, there that stall is again for another reason.
>>>>>
>>>>> I would try and tune towards what is working for you - http1 appears
>>>>> better - go with it. If you can use delete by query more sparingly, I would
>>>>> -perhaps batch them when updates are not so common, that’s a big
>>>>> instigator. Careful mixing in commits with it.
>>>>>
>>>>> You can check and make sure your server idle timeouts are higher than
>>>>> the clients - that can help a bit.
>>>>>
>>>>> If you are using the cloud client or even if you are not, you can hash
>>>>> on ids client side and send updates right to the leader they belong on to
>>>>> help reduce extraneous zig zag update traffic.
>>>>>
>>>>> In an intensive test world, this takes a large, multi pronged path to
>>>>> fully eliminate. In a production world under your control, you should be
>>>>> able to find a reasonable result with a bit of what you have been exploring
>>>>> and some acceptance for imperfection or use adjustment.
>>>>>
>>>>> Things can also change as Jetty dependencies are updated - though I
>>>>> will say in my experience not often for the better. A positive if one is
>>>>> working on development, perhaps less so in production. And again, even
>>>>> still, http 1.1 tends to be more forgiving. And delete by query and some
>>>>> other issues are still a bit more persistent, but less likely to move
>>>>> backwards or unexpectedly on you as the more ethereal connection issues and
>>>>> spurious EOF parsing exceptions.
>>>>>
>>>>> Many users are not heavily troubled by the above, so likely you will
>>>>> find a workable setup. That background is to essentially say, you may find
>>>>> clear sky’s tomorrow, but also, it’s not just some setup or use issue on
>>>>> your end, and also, while a single fix might set you up, keep in mind a
>>>>> single fix may be what your waiting for only by chance of your situation,
>>>>> so be proactive.
>>>>>
>>>>> MRM
>>>>>
>>>>> On Wed, May 19, 2021 at 8:48 AM Ding, Lehong (external - Project)
>>>>> <le...@sap.com.invalid> wrote:
>>>>>
>>>>>> *Background:*
>>>>>>
>>>>>> Before we moving to solr 8.8.1 from 7.7.2, we performed some
>>>>>> performance test on solr 8.8.1. We met a lot of concurrent update error in
>>>>>> solr log.
>>>>>>
>>>>>> *Envrironment:*
>>>>>>
>>>>>> solrCloud with 3 cluster nodes with 500 collections, 5 collections
>>>>>> have about 10m documents.
>>>>>>
>>>>>> (1 shard, 3 replica)
>>>>>>
>>>>>> *Threads:*
>>>>>>
>>>>>> 30 update/add threads + 10 deleteByQuery threads
>>>>>>
>>>>>> *Results:*
>>>>>>
>>>>>> During deleteByQuery thread runing, only one node (lead node) has
>>>>>> update transactions, but other two node has none .
>>>>>>
>>>>>> *Errrors:  *
>>>>>>
>>>>>> java.io.IOException: Request processing has stalled for 20091ms with
>>>>>> 100 remaining elements in the queue.java.io.IOException: Request processing
>>>>>> has stalled for 20091ms with 100 remaining elements in the queue. at
>>>>>> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:449)
>>>>>> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) at
>>>>>> org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:345)
>>>>>> at
>>>>>> org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:338)
>>>>>> at
>>>>>> org.apache.solr.update.SolrCmdDistributor.distribAdd(SolrCmdDistributor.java:244)
>>>>>> at
>>>>>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribAdd(DistributedZkUpdateProcessor.java:300)
>>>>>> at
>>>>>> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:230)
>>>>>> at
>>>>>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.processAdd(DistributedZkUpdateProcessor.java:245)
>>>>>> at
>>>>>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:106)
>>>>>> at
>>>>>> org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:110)
>>>>>> at
>>>>>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:343)
>>>>>> at
>>>>>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readIterator(JavaBinUpdateRequestCodec.java:291)
>>>>>> at
>>>>>> org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:338)
>>>>>> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:283)
>>>>>> at
>>>>>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readNamedList(JavaBinUpdateRequestCodec.java:244)
>>>>>>
>>>>>>
>>>>>>
>>>>>> * Temporary Solution:*
>>>>>>
>>>>>> adding -Dsolr.http1=1 in solr start parameters
>>>>>>
>>>>>> There are still some error in error log but the number is much small.
>>>>>>
>>>>>>
>>>>>>
>>>>>> *My Questions:*
>>>>>>
>>>>>> 1 We found solr cluster will eventually get the data consistent.
>>>>>> What’s the concurrent update error mainly impacted?
>>>>>>
>>>>>> 2 Adding  -Dsolr.http1=1 in solr start parameters can reduce the
>>>>>> error number. Do we realy need add this parameter? And does this parameter
>>>>>> will be kept in later version?
>>>>>>
>>>>>> Many Thanks.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks and Regards,
>>>>>>
>>>>>> Lehong Ding
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> --
>>>>> - Mark
>>>>>
>>>>> http://about.me/markrmiller
>>>>>
>>>> --
>>>> - Mark
>>>>
>>>> http://about.me/markrmiller
>>>>
>>> --
>>> - Mark
>>>
>>> http://about.me/markrmiller
>>>
>> --
> - Mark
>
> http://about.me/markrmiller
>
-- 
- Mark

http://about.me/markrmiller

Re: ConcurrentUpdate issue of solr 8.8.1

[Mark Miller <ma...@gmail.com>]

There is no line of code to point to.

I see in the stack trace now that SolrCmdDistributor was updated to the
http2 client.

Lots more stalls and exceptions seen unless that client is changed to use
http1 instead of 2.

But still a lesser problem remains with 1.1.

That is a bit of info on why this would be possible by switching from the
Apache client to the Jetty client. And why Http2 can be worse than one.

Poor connection management. JavaBin additional aggravation.

That’s not a line of code. Probably not even a single Jira issue.

MRM

On Wed, Jun 9, 2021 at 8:47 AM David Smiley <ds...@apache.org> wrote:

> Mark, can you file a JIRA issue (if one doesn't already exist), ideally
> pointing to a line of code that is problematic?
>
> Do you think we should drop the "closeShield" stuff in
> SolrDispatchFilter?  Not sure if there is a test exercising the scenario
> that led to its inclusion; I doubt it.
>
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Wed, Jun 9, 2021 at 12:25 AM Mark Miller <ma...@gmail.com> wrote:
>
>> A few details on how JavaBin can be involved in this type of behavior:
>>
>> JavaBin consumes data by by wrapping InputStream’s with a
>> FastInputStream. FastInputStream implements an InputStream, but it
>> disregards the InputStream contract that says you have reached the end of
>> the stream when -1 is returned. A -1 is just another bit of data to be
>> shifted into a byte and either gum the works or corrupt the type or size
>> read.
>>
>> Jetty does not give you a raw InputStream to the network that that will
>> simply suck up data like a file from disk. You will get something different
>> depending on the version, depending on the client, depending on http1 or 2
>> in some cases, at least in terms of behavior.
>>
>> Generally you are going to get something backed by ByteBuffers pretending
>> to be a raw stream.
>>
>> Depending on different circumstances, Jetty will decide to stop that
>> stream by jumping in with a -1 when it hits circumstances that lead to
>> killing the connection.
>>
>> The specifics for when and why will vary between http1 and 2. Http1 will
>> often be using chunked encoding and have little idea about the the scope of
>> the full content. It knows the size of each chunk. Http2 has no such
>> encoding and works with sessions and frames and an entirely different
>> protocol and needs to stop a channel or session. Multiplexing means this
>> can affect things across requests in a way that is different than Http1. In
>> Solr, you may also have different clients depending on the protocol with
>> different retry implementations. Old Jetty clients offered retry, their
>> view for the newer client is that it’s too application specific as an
>> option, implement retry yourself if you want it. The Apache http1 client
>> will retry based on the Solr retry configuration class.
>>
>> Given that JavaBin has little respect for streams and deals in expected
>> sizes or it’s own end markers, there is plenty of room to not clear a
>> request. Given we try and not close streams due to old webapp days of
>> tomcat and earlier jetty versions, there is more room for poor behavior.
>> Not closing streams is no longer helpful at all in most cases. Generally
>> there is an attempt to read to end of stream, but this is a hit ir miss
>> strategy, especially as currently done.
>>
>> Instead of respecting stream contracts, JavaBin is given how much to read
>> and it diligently works to read to that point or in some cases it’s own end
>> marker.
>>
>> So when Jetty can’t tell a stream is actually complete, it needs to
>> timeout before it will forcibly kill it. Varying behavior between http 1
>> and 2. For various reasons it may try to do this as for other reasons as
>> well. That will cause negative affects (delays in many cases) and reset or
>> closed exceptions. The -1 it will inject into the stream won’t generally
>> get a prompt response either. Depending on timing and data, JavaBin may not
>> even see this an early end and simply spit out something off.  Retries from
>> the same client won’t help for some time in some cases (HTTP2 worse than 1)
>> either. Mileage will vary depending on HTTP2 or 1.1 for most of this. Http2
>> can send things like GoAway or session lifecycle info as well that will
>> lead to poorly handled exceptions in a similar vein.
>>
>> There are other things that could be at play, but this is one that
>> behaves worse with http2 and also appears worse with Jetty 10 / 11 than 9.
>> The better the system acts, the more visible these problems tend to be.
>>
>> MRM
>>
>> On Fri, Jun 4, 2021 at 3:55 PM Mark Miller <ma...@gmail.com> wrote:
>>
>>> Given you say you saw a lot of this and that occurred after a specific
>>> upgrade, I would guess your root issue is JavaBin.
>>>
>>> Unfortunately, I don’t believe you can change internode communication to
>>> another format if that’s the case.
>>>
>>> But also if that is the case, it will be in the way of some work I have
>>> to do over the coming months and so I will fix it.
>>>
>>> Mark
>>>
>>> On Wed, May 26, 2021 at 7:46 PM Mark Miller <ma...@gmail.com>
>>> wrote:
>>>
>>>> You can run into this fairly easily even if you are doing everything
>>>> right and none of your operations are intrinsically slow or problematic.
>>>>
>>>> DeleteByQuery, regardless of its cost, tends to be a large contributor,
>>>> though not always. You can mitigate a bit with cautious, controlled use of
>>>> it.
>>>>
>>>> I’m not surprised that http2 is even more prone to also being involved,
>>>> though I didn’t think that client was yet using an http2 version, so that
>>>> is a bit surprising, but these things can bleed over quite easily. Even
>>>> more so in new Jetty versions.
>>>>
>>>> Jetty client -> server communication (and http2 in general) can be much
>>>> more picky around handling connections that are not tightly managed for
>>>> resuse under http2 (which can multiplex many requests over a single
>>>> connection). If you don’t fully read input/output streams for example, the
>>>> server doesn’t know that you don’t intend to finish dealing with your
>>>> stream. It will wait some amount of time. And then it will whack that
>>>> connection. There are all sorts of things that can manifest from this
>>>> depending on all kinds of factors, but one of them is your client server
>>>> communication can be hosed for a bit. Similarity things can happen even if
>>>> you do always keep your connection pool connections in shape if say, you
>>>> set a content length header that doesn’t match the content. You can do a
>>>> lot poorly all day and hardly notice a peep unless you turn on debug
>>>> logging for jetty or monitor tcp stats. And most of the time, things won’t
>>>> be terrible as a result of it either. But every now and then you get pretty
>>>> annoying consequences. And if you have something aggravating in the mix,
>>>> maybe more than now and then.
>>>>
>>>> As I said though, you can run into this dist stall issue from a variety
>>>> of ways, you can march down the list of what can cause it and low and
>>>> behold, there that stall is again for another reason.
>>>>
>>>> I would try and tune towards what is working for you - http1 appears
>>>> better - go with it. If you can use delete by query more sparingly, I would
>>>> -perhaps batch them when updates are not so common, that’s a big
>>>> instigator. Careful mixing in commits with it.
>>>>
>>>> You can check and make sure your server idle timeouts are higher than
>>>> the clients - that can help a bit.
>>>>
>>>> If you are using the cloud client or even if you are not, you can hash
>>>> on ids client side and send updates right to the leader they belong on to
>>>> help reduce extraneous zig zag update traffic.
>>>>
>>>> In an intensive test world, this takes a large, multi pronged path to
>>>> fully eliminate. In a production world under your control, you should be
>>>> able to find a reasonable result with a bit of what you have been exploring
>>>> and some acceptance for imperfection or use adjustment.
>>>>
>>>> Things can also change as Jetty dependencies are updated - though I
>>>> will say in my experience not often for the better. A positive if one is
>>>> working on development, perhaps less so in production. And again, even
>>>> still, http 1.1 tends to be more forgiving. And delete by query and some
>>>> other issues are still a bit more persistent, but less likely to move
>>>> backwards or unexpectedly on you as the more ethereal connection issues and
>>>> spurious EOF parsing exceptions.
>>>>
>>>> Many users are not heavily troubled by the above, so likely you will
>>>> find a workable setup. That background is to essentially say, you may find
>>>> clear sky’s tomorrow, but also, it’s not just some setup or use issue on
>>>> your end, and also, while a single fix might set you up, keep in mind a
>>>> single fix may be what your waiting for only by chance of your situation,
>>>> so be proactive.
>>>>
>>>> MRM
>>>>
>>>> On Wed, May 19, 2021 at 8:48 AM Ding, Lehong (external - Project)
>>>> <le...@sap.com.invalid> wrote:
>>>>
>>>>> *Background:*
>>>>>
>>>>> Before we moving to solr 8.8.1 from 7.7.2, we performed some
>>>>> performance test on solr 8.8.1. We met a lot of concurrent update error in
>>>>> solr log.
>>>>>
>>>>> *Envrironment:*
>>>>>
>>>>> solrCloud with 3 cluster nodes with 500 collections, 5 collections
>>>>> have about 10m documents.
>>>>>
>>>>> (1 shard, 3 replica)
>>>>>
>>>>> *Threads:*
>>>>>
>>>>> 30 update/add threads + 10 deleteByQuery threads
>>>>>
>>>>> *Results:*
>>>>>
>>>>> During deleteByQuery thread runing, only one node (lead node) has
>>>>> update transactions, but other two node has none .
>>>>>
>>>>> *Errrors:  *
>>>>>
>>>>> java.io.IOException: Request processing has stalled for 20091ms with
>>>>> 100 remaining elements in the queue.java.io.IOException: Request processing
>>>>> has stalled for 20091ms with 100 remaining elements in the queue. at
>>>>> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:449)
>>>>> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) at
>>>>> org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:345)
>>>>> at
>>>>> org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:338)
>>>>> at
>>>>> org.apache.solr.update.SolrCmdDistributor.distribAdd(SolrCmdDistributor.java:244)
>>>>> at
>>>>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribAdd(DistributedZkUpdateProcessor.java:300)
>>>>> at
>>>>> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:230)
>>>>> at
>>>>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.processAdd(DistributedZkUpdateProcessor.java:245)
>>>>> at
>>>>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:106)
>>>>> at
>>>>> org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:110)
>>>>> at
>>>>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:343)
>>>>> at
>>>>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readIterator(JavaBinUpdateRequestCodec.java:291)
>>>>> at
>>>>> org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:338)
>>>>> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:283)
>>>>> at
>>>>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readNamedList(JavaBinUpdateRequestCodec.java:244)
>>>>>
>>>>>
>>>>>
>>>>> * Temporary Solution:*
>>>>>
>>>>> adding -Dsolr.http1=1 in solr start parameters
>>>>>
>>>>> There are still some error in error log but the number is much small.
>>>>>
>>>>>
>>>>>
>>>>> *My Questions:*
>>>>>
>>>>> 1 We found solr cluster will eventually get the data consistent.
>>>>> What’s the concurrent update error mainly impacted?
>>>>>
>>>>> 2 Adding  -Dsolr.http1=1 in solr start parameters can reduce the error
>>>>> number. Do we realy need add this parameter? And does this parameter will
>>>>> be kept in later version?
>>>>>
>>>>> Many Thanks.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thanks and Regards,
>>>>>
>>>>> Lehong Ding
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> --
>>>> - Mark
>>>>
>>>> http://about.me/markrmiller
>>>>
>>> --
>>> - Mark
>>>
>>> http://about.me/markrmiller
>>>
>> --
>> - Mark
>>
>> http://about.me/markrmiller
>>
> --
- Mark

http://about.me/markrmiller

Re: ConcurrentUpdate issue of solr 8.8.1

[David Smiley <ds...@apache.org>]

Mark, can you file a JIRA issue (if one doesn't already exist), ideally
pointing to a line of code that is problematic?

Do you think we should drop the "closeShield" stuff in SolrDispatchFilter?
Not sure if there is a test exercising the scenario that led to its
inclusion; I doubt it.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Wed, Jun 9, 2021 at 12:25 AM Mark Miller <ma...@gmail.com> wrote:

> A few details on how JavaBin can be involved in this type of behavior:
>
> JavaBin consumes data by by wrapping InputStream’s with a FastInputStream.
> FastInputStream implements an InputStream, but it disregards the
> InputStream contract that says you have reached the end of the stream when
> -1 is returned. A -1 is just another bit of data to be shifted into a byte
> and either gum the works or corrupt the type or size read.
>
> Jetty does not give you a raw InputStream to the network that that will
> simply suck up data like a file from disk. You will get something different
> depending on the version, depending on the client, depending on http1 or 2
> in some cases, at least in terms of behavior.
>
> Generally you are going to get something backed by ByteBuffers pretending
> to be a raw stream.
>
> Depending on different circumstances, Jetty will decide to stop that
> stream by jumping in with a -1 when it hits circumstances that lead to
> killing the connection.
>
> The specifics for when and why will vary between http1 and 2. Http1 will
> often be using chunked encoding and have little idea about the the scope of
> the full content. It knows the size of each chunk. Http2 has no such
> encoding and works with sessions and frames and an entirely different
> protocol and needs to stop a channel or session. Multiplexing means this
> can affect things across requests in a way that is different than Http1. In
> Solr, you may also have different clients depending on the protocol with
> different retry implementations. Old Jetty clients offered retry, their
> view for the newer client is that it’s too application specific as an
> option, implement retry yourself if you want it. The Apache http1 client
> will retry based on the Solr retry configuration class.
>
> Given that JavaBin has little respect for streams and deals in expected
> sizes or it’s own end markers, there is plenty of room to not clear a
> request. Given we try and not close streams due to old webapp days of
> tomcat and earlier jetty versions, there is more room for poor behavior.
> Not closing streams is no longer helpful at all in most cases. Generally
> there is an attempt to read to end of stream, but this is a hit ir miss
> strategy, especially as currently done.
>
> Instead of respecting stream contracts, JavaBin is given how much to read
> and it diligently works to read to that point or in some cases it’s own end
> marker.
>
> So when Jetty can’t tell a stream is actually complete, it needs to
> timeout before it will forcibly kill it. Varying behavior between http 1
> and 2. For various reasons it may try to do this as for other reasons as
> well. That will cause negative affects (delays in many cases) and reset or
> closed exceptions. The -1 it will inject into the stream won’t generally
> get a prompt response either. Depending on timing and data, JavaBin may not
> even see this an early end and simply spit out something off.  Retries from
> the same client won’t help for some time in some cases (HTTP2 worse than 1)
> either. Mileage will vary depending on HTTP2 or 1.1 for most of this. Http2
> can send things like GoAway or session lifecycle info as well that will
> lead to poorly handled exceptions in a similar vein.
>
> There are other things that could be at play, but this is one that behaves
> worse with http2 and also appears worse with Jetty 10 / 11 than 9. The
> better the system acts, the more visible these problems tend to be.
>
> MRM
>
> On Fri, Jun 4, 2021 at 3:55 PM Mark Miller <ma...@gmail.com> wrote:
>
>> Given you say you saw a lot of this and that occurred after a specific
>> upgrade, I would guess your root issue is JavaBin.
>>
>> Unfortunately, I don’t believe you can change internode communication to
>> another format if that’s the case.
>>
>> But also if that is the case, it will be in the way of some work I have
>> to do over the coming months and so I will fix it.
>>
>> Mark
>>
>> On Wed, May 26, 2021 at 7:46 PM Mark Miller <ma...@gmail.com>
>> wrote:
>>
>>> You can run into this fairly easily even if you are doing everything
>>> right and none of your operations are intrinsically slow or problematic.
>>>
>>> DeleteByQuery, regardless of its cost, tends to be a large contributor,
>>> though not always. You can mitigate a bit with cautious, controlled use of
>>> it.
>>>
>>> I’m not surprised that http2 is even more prone to also being involved,
>>> though I didn’t think that client was yet using an http2 version, so that
>>> is a bit surprising, but these things can bleed over quite easily. Even
>>> more so in new Jetty versions.
>>>
>>> Jetty client -> server communication (and http2 in general) can be much
>>> more picky around handling connections that are not tightly managed for
>>> resuse under http2 (which can multiplex many requests over a single
>>> connection). If you don’t fully read input/output streams for example, the
>>> server doesn’t know that you don’t intend to finish dealing with your
>>> stream. It will wait some amount of time. And then it will whack that
>>> connection. There are all sorts of things that can manifest from this
>>> depending on all kinds of factors, but one of them is your client server
>>> communication can be hosed for a bit. Similarity things can happen even if
>>> you do always keep your connection pool connections in shape if say, you
>>> set a content length header that doesn’t match the content. You can do a
>>> lot poorly all day and hardly notice a peep unless you turn on debug
>>> logging for jetty or monitor tcp stats. And most of the time, things won’t
>>> be terrible as a result of it either. But every now and then you get pretty
>>> annoying consequences. And if you have something aggravating in the mix,
>>> maybe more than now and then.
>>>
>>> As I said though, you can run into this dist stall issue from a variety
>>> of ways, you can march down the list of what can cause it and low and
>>> behold, there that stall is again for another reason.
>>>
>>> I would try and tune towards what is working for you - http1 appears
>>> better - go with it. If you can use delete by query more sparingly, I would
>>> -perhaps batch them when updates are not so common, that’s a big
>>> instigator. Careful mixing in commits with it.
>>>
>>> You can check and make sure your server idle timeouts are higher than
>>> the clients - that can help a bit.
>>>
>>> If you are using the cloud client or even if you are not, you can hash
>>> on ids client side and send updates right to the leader they belong on to
>>> help reduce extraneous zig zag update traffic.
>>>
>>> In an intensive test world, this takes a large, multi pronged path to
>>> fully eliminate. In a production world under your control, you should be
>>> able to find a reasonable result with a bit of what you have been exploring
>>> and some acceptance for imperfection or use adjustment.
>>>
>>> Things can also change as Jetty dependencies are updated - though I will
>>> say in my experience not often for the better. A positive if one is working
>>> on development, perhaps less so in production. And again, even still, http
>>> 1.1 tends to be more forgiving. And delete by query and some other issues
>>> are still a bit more persistent, but less likely to move backwards or
>>> unexpectedly on you as the more ethereal connection issues and spurious EOF
>>> parsing exceptions.
>>>
>>> Many users are not heavily troubled by the above, so likely you will
>>> find a workable setup. That background is to essentially say, you may find
>>> clear sky’s tomorrow, but also, it’s not just some setup or use issue on
>>> your end, and also, while a single fix might set you up, keep in mind a
>>> single fix may be what your waiting for only by chance of your situation,
>>> so be proactive.
>>>
>>> MRM
>>>
>>> On Wed, May 19, 2021 at 8:48 AM Ding, Lehong (external - Project)
>>> <le...@sap.com.invalid> wrote:
>>>
>>>> *Background:*
>>>>
>>>> Before we moving to solr 8.8.1 from 7.7.2, we performed some
>>>> performance test on solr 8.8.1. We met a lot of concurrent update error in
>>>> solr log.
>>>>
>>>> *Envrironment:*
>>>>
>>>> solrCloud with 3 cluster nodes with 500 collections, 5 collections have
>>>> about 10m documents.
>>>>
>>>> (1 shard, 3 replica)
>>>>
>>>> *Threads:*
>>>>
>>>> 30 update/add threads + 10 deleteByQuery threads
>>>>
>>>> *Results:*
>>>>
>>>> During deleteByQuery thread runing, only one node (lead node) has
>>>> update transactions, but other two node has none .
>>>>
>>>> *Errrors:  *
>>>>
>>>> java.io.IOException: Request processing has stalled for 20091ms with
>>>> 100 remaining elements in the queue.java.io.IOException: Request processing
>>>> has stalled for 20091ms with 100 remaining elements in the queue. at
>>>> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:449)
>>>> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) at
>>>> org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:345)
>>>> at
>>>> org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:338)
>>>> at
>>>> org.apache.solr.update.SolrCmdDistributor.distribAdd(SolrCmdDistributor.java:244)
>>>> at
>>>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribAdd(DistributedZkUpdateProcessor.java:300)
>>>> at
>>>> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:230)
>>>> at
>>>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.processAdd(DistributedZkUpdateProcessor.java:245)
>>>> at
>>>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:106)
>>>> at
>>>> org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:110)
>>>> at
>>>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:343)
>>>> at
>>>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readIterator(JavaBinUpdateRequestCodec.java:291)
>>>> at
>>>> org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:338)
>>>> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:283)
>>>> at
>>>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readNamedList(JavaBinUpdateRequestCodec.java:244)
>>>>
>>>>
>>>>
>>>> * Temporary Solution:*
>>>>
>>>> adding -Dsolr.http1=1 in solr start parameters
>>>>
>>>> There are still some error in error log but the number is much small.
>>>>
>>>>
>>>>
>>>> *My Questions:*
>>>>
>>>> 1 We found solr cluster will eventually get the data consistent. What’s
>>>> the concurrent update error mainly impacted?
>>>>
>>>> 2 Adding  -Dsolr.http1=1 in solr start parameters can reduce the error
>>>> number. Do we realy need add this parameter? And does this parameter will
>>>> be kept in later version?
>>>>
>>>> Many Thanks.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Thanks and Regards,
>>>>
>>>> Lehong Ding
>>>>
>>>>
>>>>
>>>>
>>>>
>>> --
>>> - Mark
>>>
>>> http://about.me/markrmiller
>>>
>> --
>> - Mark
>>
>> http://about.me/markrmiller
>>
> --
> - Mark
>
> http://about.me/markrmiller
>

Re: ConcurrentUpdate issue of solr 8.8.1

[Mark Miller <ma...@gmail.com>]

A few details on how JavaBin can be involved in this type of behavior:

JavaBin consumes data by by wrapping InputStream’s with a FastInputStream.
FastInputStream implements an InputStream, but it disregards the
InputStream contract that says you have reached the end of the stream when
-1 is returned. A -1 is just another bit of data to be shifted into a byte
and either gum the works or corrupt the type or size read.

Jetty does not give you a raw InputStream to the network that that will
simply suck up data like a file from disk. You will get something different
depending on the version, depending on the client, depending on http1 or 2
in some cases, at least in terms of behavior.

Generally you are going to get something backed by ByteBuffers pretending
to be a raw stream.

Depending on different circumstances, Jetty will decide to stop that stream
by jumping in with a -1 when it hits circumstances that lead to killing the
connection.

The specifics for when and why will vary between http1 and 2. Http1 will
often be using chunked encoding and have little idea about the the scope of
the full content. It knows the size of each chunk. Http2 has no such
encoding and works with sessions and frames and an entirely different
protocol and needs to stop a channel or session. Multiplexing means this
can affect things across requests in a way that is different than Http1. In
Solr, you may also have different clients depending on the protocol with
different retry implementations. Old Jetty clients offered retry, their
view for the newer client is that it’s too application specific as an
option, implement retry yourself if you want it. The Apache http1 client
will retry based on the Solr retry configuration class.

Given that JavaBin has little respect for streams and deals in expected
sizes or it’s own end markers, there is plenty of room to not clear a
request. Given we try and not close streams due to old webapp days of
tomcat and earlier jetty versions, there is more room for poor behavior.
Not closing streams is no longer helpful at all in most cases. Generally
there is an attempt to read to end of stream, but this is a hit ir miss
strategy, especially as currently done.

Instead of respecting stream contracts, JavaBin is given how much to read
and it diligently works to read to that point or in some cases it’s own end
marker.

So when Jetty can’t tell a stream is actually complete, it needs to timeout
before it will forcibly kill it. Varying behavior between http 1 and 2. For
various reasons it may try to do this as for other reasons as well. That
will cause negative affects (delays in many cases) and reset or closed
exceptions. The -1 it will inject into the stream won’t generally get a
prompt response either. Depending on timing and data, JavaBin may not even
see this an early end and simply spit out something off.  Retries from the
same client won’t help for some time in some cases (HTTP2 worse than 1)
either. Mileage will vary depending on HTTP2 or 1.1 for most of this. Http2
can send things like GoAway or session lifecycle info as well that will
lead to poorly handled exceptions in a similar vein.

There are other things that could be at play, but this is one that behaves
worse with http2 and also appears worse with Jetty 10 / 11 than 9. The
better the system acts, the more visible these problems tend to be.

MRM

On Fri, Jun 4, 2021 at 3:55 PM Mark Miller <ma...@gmail.com> wrote:

> Given you say you saw a lot of this and that occurred after a specific
> upgrade, I would guess your root issue is JavaBin.
>
> Unfortunately, I don’t believe you can change internode communication to
> another format if that’s the case.
>
> But also if that is the case, it will be in the way of some work I have to
> do over the coming months and so I will fix it.
>
> Mark
>
> On Wed, May 26, 2021 at 7:46 PM Mark Miller <ma...@gmail.com> wrote:
>
>> You can run into this fairly easily even if you are doing everything
>> right and none of your operations are intrinsically slow or problematic.
>>
>> DeleteByQuery, regardless of its cost, tends to be a large contributor,
>> though not always. You can mitigate a bit with cautious, controlled use of
>> it.
>>
>> I’m not surprised that http2 is even more prone to also being involved,
>> though I didn’t think that client was yet using an http2 version, so that
>> is a bit surprising, but these things can bleed over quite easily. Even
>> more so in new Jetty versions.
>>
>> Jetty client -> server communication (and http2 in general) can be much
>> more picky around handling connections that are not tightly managed for
>> resuse under http2 (which can multiplex many requests over a single
>> connection). If you don’t fully read input/output streams for example, the
>> server doesn’t know that you don’t intend to finish dealing with your
>> stream. It will wait some amount of time. And then it will whack that
>> connection. There are all sorts of things that can manifest from this
>> depending on all kinds of factors, but one of them is your client server
>> communication can be hosed for a bit. Similarity things can happen even if
>> you do always keep your connection pool connections in shape if say, you
>> set a content length header that doesn’t match the content. You can do a
>> lot poorly all day and hardly notice a peep unless you turn on debug
>> logging for jetty or monitor tcp stats. And most of the time, things won’t
>> be terrible as a result of it either. But every now and then you get pretty
>> annoying consequences. And if you have something aggravating in the mix,
>> maybe more than now and then.
>>
>> As I said though, you can run into this dist stall issue from a variety
>> of ways, you can march down the list of what can cause it and low and
>> behold, there that stall is again for another reason.
>>
>> I would try and tune towards what is working for you - http1 appears
>> better - go with it. If you can use delete by query more sparingly, I would
>> -perhaps batch them when updates are not so common, that’s a big
>> instigator. Careful mixing in commits with it.
>>
>> You can check and make sure your server idle timeouts are higher than the
>> clients - that can help a bit.
>>
>> If you are using the cloud client or even if you are not, you can hash on
>> ids client side and send updates right to the leader they belong on to help
>> reduce extraneous zig zag update traffic.
>>
>> In an intensive test world, this takes a large, multi pronged path to
>> fully eliminate. In a production world under your control, you should be
>> able to find a reasonable result with a bit of what you have been exploring
>> and some acceptance for imperfection or use adjustment.
>>
>> Things can also change as Jetty dependencies are updated - though I will
>> say in my experience not often for the better. A positive if one is working
>> on development, perhaps less so in production. And again, even still, http
>> 1.1 tends to be more forgiving. And delete by query and some other issues
>> are still a bit more persistent, but less likely to move backwards or
>> unexpectedly on you as the more ethereal connection issues and spurious EOF
>> parsing exceptions.
>>
>> Many users are not heavily troubled by the above, so likely you will find
>> a workable setup. That background is to essentially say, you may find clear
>> sky’s tomorrow, but also, it’s not just some setup or use issue on your
>> end, and also, while a single fix might set you up, keep in mind a single
>> fix may be what your waiting for only by chance of your situation, so be
>> proactive.
>>
>> MRM
>>
>> On Wed, May 19, 2021 at 8:48 AM Ding, Lehong (external - Project)
>> <le...@sap.com.invalid> wrote:
>>
>>> *Background:*
>>>
>>> Before we moving to solr 8.8.1 from 7.7.2, we performed some performance
>>> test on solr 8.8.1. We met a lot of concurrent update error in solr log.
>>>
>>> *Envrironment:*
>>>
>>> solrCloud with 3 cluster nodes with 500 collections, 5 collections have
>>> about 10m documents.
>>>
>>> (1 shard, 3 replica)
>>>
>>> *Threads:*
>>>
>>> 30 update/add threads + 10 deleteByQuery threads
>>>
>>> *Results:*
>>>
>>> During deleteByQuery thread runing, only one node (lead node) has update
>>> transactions, but other two node has none .
>>>
>>> *Errrors:  *
>>>
>>> java.io.IOException: Request processing has stalled for 20091ms with 100
>>> remaining elements in the queue.java.io.IOException: Request processing has
>>> stalled for 20091ms with 100 remaining elements in the queue. at
>>> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:449)
>>> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) at
>>> org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:345)
>>> at
>>> org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:338)
>>> at
>>> org.apache.solr.update.SolrCmdDistributor.distribAdd(SolrCmdDistributor.java:244)
>>> at
>>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribAdd(DistributedZkUpdateProcessor.java:300)
>>> at
>>> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:230)
>>> at
>>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.processAdd(DistributedZkUpdateProcessor.java:245)
>>> at
>>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:106)
>>> at
>>> org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:110)
>>> at
>>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:343)
>>> at
>>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readIterator(JavaBinUpdateRequestCodec.java:291)
>>> at
>>> org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:338)
>>> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:283)
>>> at
>>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readNamedList(JavaBinUpdateRequestCodec.java:244)
>>>
>>>
>>>
>>> * Temporary Solution:*
>>>
>>> adding -Dsolr.http1=1 in solr start parameters
>>>
>>> There are still some error in error log but the number is much small.
>>>
>>>
>>>
>>> *My Questions:*
>>>
>>> 1 We found solr cluster will eventually get the data consistent. What’s
>>> the concurrent update error mainly impacted?
>>>
>>> 2 Adding  -Dsolr.http1=1 in solr start parameters can reduce the error
>>> number. Do we realy need add this parameter? And does this parameter will
>>> be kept in later version?
>>>
>>> Many Thanks.
>>>
>>>
>>>
>>>
>>>
>>> Thanks and Regards,
>>>
>>> Lehong Ding
>>>
>>>
>>>
>>>
>>>
>> --
>> - Mark
>>
>> http://about.me/markrmiller
>>
> --
> - Mark
>
> http://about.me/markrmiller
>
-- 
- Mark

http://about.me/markrmiller