[jira] [Created] (SOLR-18087) HTTP/2 Struggles With Streaming Large Responses

["Luke Kot-Zaniewski (Jira)" <ji...@apache.org>]

Luke Kot-Zaniewski created SOLR-18087:
-----------------------------------------

             Summary: HTTP/2 Struggles With Streaming Large Responses
                 Key: SOLR-18087
                 URL: https://issues.apache.org/jira/browse/SOLR-18087
             Project: Solr
          Issue Type: Bug
            Reporter: Luke Kot-Zaniewski
         Attachments: flow-control-stall.log, index-recovery-tests.md, stream-benchmark-results.md

There appear to be some severe http/2 regressions since at least 9.8, most notably with the stream handler as well as index recovery. The impact is at the very least slowness and in some cases outright response stalling. The response stalling appears to be caused by HTTP/2's flow control. The obvious thing these two very different workloads share in common is that they stream large responses. This means, among other things, that they may be more directly impacted by HTTP2's flow control mechanism.

In my testing I have tweaked the following parameters:
 # http1 vs http2 - as stated, http1 seems to be strictly better as in faster and more stable.
 # shards per node - the greater the number of shards per node the more (large, simultaneous) responses share a single connection during inter-node communication. This has generally resulted in poorer performance.
 #  maxConcurrentStreams - reducing this to, say 1, can effectively circumvent multiplexing. Circumventing multiplexing does seem to improve index recovery somewhat (still slower than HTTP/1). On the other hand, this seems antithetical to the point of http2. It's also interesting this doesn't help 
 #  initialSessionRecvWindow - This is the amount of buffer the client gets for each connection. This gets shared by the many responses that share the multiplexed connection.
 #  initialStreamRecvWindow - This is the amount of buffer each stream gets within a single HTTP/2 session. I've found that when this is too big relative to initialSessionRecvWindow it can lead to stalling because of flow control enforcement


I’m attaching summaries of my findings, some of which can be reproduced by running the appropriate benchmark in this branch: [https://github.com/kotman12/solr/tree/http2-shenanigans.]

My next step is to solicit some feedback from the community. Absent anything else I may try reproducing this in a pure jetty example. I am beginning to think multiple large responses getting streamed simultaneously between the same client and server may some kind of edge case the library doesn't handle well. It may have something to do with how Jetty's InputStreamResponseListener is implemented although according to the docs it _should_ be compatible with HTTP/2.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org