You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "Sebastian Hofmann (Jira)" <ji...@apache.org> on 2024/10/18 13:51:00 UTC
[jira] [Updated] (SOLR-17502) request forwarding does not work with http/2
[ https://issues.apache.org/jira/browse/SOLR-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sebastian Hofmann updated SOLR-17502:
-------------------------------------
Summary: request forwarding does not work with http/2 (was: overlay request forwarding does not work with http/2)
> request forwarding does not work with http/2
> --------------------------------------------
>
> Key: SOLR-17502
> URL: https://issues.apache.org/jira/browse/SOLR-17502
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
> Affects Versions: 9.7
> Reporter: Sebastian Hofmann
> Priority: Major
>
> It seems like that requests using http/2 to solr/\{core}/config/overlay (maybe also others, i only tested these) which are forwarded from one Solr node to another in a cloud setup don´t work.
>
> The exact error message looks like this:
> {code:java}
> 2024-10-17 10:26:05.536 ERROR (qtp1677458155-20-null-79) [c: s: r: x: t:null-79] o.a.s.s.HttpSolrCall 500 Exception => org.apache.solr.common.SolrException: Error trying to proxy request for url: http://172.20.0.3:8983/solr/Test2/config/overlay with _forwardCount: 1
> solr-1 | at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:817)
> solr-1 | org.apache.solr.common.SolrException: Error trying to proxy request for url: http://172.20.0.3:8983/solr/Test2/config/overlay with _forwardCount: 1
> solr-1 | at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:817) ~[?:?]
> solr-1 | at org.apache.solr.servlet.HttpSolrCall.sendRemoteQuery(HttpSolrCall.java:504) ~[?:?]
> solr-1 | at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559) ~[?:?]
> solr-1 | at org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:251) ~[?:?]
> solr-1 | at org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:208) ~[?:?]
> solr-1 | at org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:243) ~[?:?]
> solr-1 | at org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:213) ~[?:?]
> solr-1 | at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:202) ~[?:?]
> solr-1 | at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184) ~[?:?]
> solr-1 | at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210) ~[jetty-servlet-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) ~[jetty-servlet-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598) ~[jetty-security-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) ~[jetty-servlet-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:228) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301) ~[jetty-rewrite-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.Server.handle(Server.java:563) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193) ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:208) ~[http2-common-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.http2.HTTP2Connection.offerTask(HTTP2Connection.java:193) ~[http2-common-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.http2.server.HTTP2ServerConnection.onNewStream(HTTP2ServerConnection.java:144) ~[http2-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onNewStream(HTTP2ServerConnectionFactory.java:97) ~[http2-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.http2.HTTP2Session.notifyNewStream(HTTP2Session.java:1136) ~[http2-common-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.http2.server.HTTP2ServerSession.onHeaders(HTTP2ServerSession.java:120) ~[http2-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.http2.server.HTTP2ServerSession.onFrame(HTTP2ServerSession.java:188) ~[http2-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.http2.server.HTTP2ServerConnection.onOpen(HTTP2ServerConnection.java:113) ~[http2-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.io.AbstractEndPoint.upgrade(AbstractEndPoint.java:451) ~[jetty-io-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.HttpConnection.upgrade(HttpConnection.java:424) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.HttpConnection.onCompleted(HttpConnection.java:455) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.HttpChannelOverHttp.upgrade(HttpChannelOverHttp.java:677) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.HttpChannelOverHttp.headerComplete(HttpChannelOverHttp.java:416) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.http.HttpParser.parseFields(HttpParser.java:1256) ~[jetty-http-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1542) ~[jetty-http-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.HttpConnection.parseRequestBuffer(HttpConnection.java:403) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:275) ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) ~[jetty-io-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) ~[jetty-io-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) ~[jetty-io-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199) ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at java.base/java.lang.Thread.run(Unknown Source) [?:?]
> solr-1 | Caused by: org.apache.http.client.ClientProtocolException
> solr-1 | at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187) ~[?:?]
> solr-1 | at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[?:?]
> solr-1 | at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[?:?]
> solr-1 | at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:780) ~[?:?]
> solr-1 | ... 70 more
> solr-1 | Caused by: org.apache.http.ProtocolException: The server failed to respond with a valid HTTP response
> solr-1 | at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:149) ~[?:?]
> solr-1 | at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[?:?]
> solr-1 | at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[?:?]
> solr-1 | at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[?:?]
> solr-1 | at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) ~[?:?]
> solr-1 | at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[?:?]
> solr-1 | at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[?:?]
> solr-1 | at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:144) ~[?:?]
> solr-1 | at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[?:?]
> solr-1 | at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[?:?]
> solr-1 | at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[?:?]
> solr-1 | at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[?:?]
> solr-1 | at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[?:?]
> solr-1 | at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[?:?]
> solr-1 | at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[?:?]
> solr-1 | at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:780) ~[?:?]
> solr-1 | ... 70 more
> {code}
> I created a small docker-compose.yml file to reproduce it:
> {code:java}
> version: '3'
> services:
> solr:
> image: solr:9.7
> ports:
> - "8983:8983"
> - "5005:5005"
> environment:
> ZK_HOST: "zoo:2181"
> depends_on: [zoo]
> solr2:
> image: solr:9.7
> ports:
> - "8984:8983"
> environment:
> ZK_HOST: "zoo:2181"
> depends_on: [zoo]
> zoo:
> image: zookeeper:3.9
> ports:
> - "2181:2181"
> environment:
> ZOO_4LW_COMMANDS_WHITELIST: "mntr,conf,ruok" {code}
> I created two collections using this curl commands:
> {code:java}
> curl "http://localhost:8983/solr/admin/collections?&action=CREATE&collection.configName=_default&name=Test&numShards=1&replicationFactor=1&router.name=compositeId&wt=json"
> curl "http://localhost:8983/solr/admin/collections?&action=CREATE&collection.configName=_default&name=Test2&numShards=1&replicationFactor=1&router.name=compositeId&wt=json" {code}
> In my case the collections were stored each on one node. I can send requests and solr forwards them to the right node:
> {code:java}
> curl "http://localhost:8983/solr/Test/config/overlay"
> {
> "responseHeader":{
> "status":0,
> "QTime":28
> },
> "overlay":{
> "znodeVersion":-1
> }
> }%
> curl "http://localhost:8983/solr/Test2/config/overlay"
> {
> "responseHeader":{
> "status":0,
> "QTime":25
> },
> "overlay":{
> "znodeVersion":-1
> }
> }% {code}
> Everything works fine, untill http/2 comes in to play and the server needs to forward the request, then the Request fails:
> {code:java}
> curl --http2 "http://localhost:8983/solr/Test2/config/overlay"
> curl: (92) HTTP/2 stream 1 was not closed cleanly: CANCEL (err 8)
> curl --http2 "http://localhost:8983/solr/Test/config/overlay"
> {
> "responseHeader":{
> "status":0,
> "QTime":2
> },
> "overlay":{
> "znodeVersion":-1
> }
> }
> curl --http2 "http://localhost:8984/solr/Test/config/overlay"
> curl: (92) HTTP/2 stream 1 was not closed cleanly: CANCEL (err 8)
> curl --http2 "http://localhost:8984/solr/Test2/config/overlay"
> {
> "responseHeader":{
> "status":0,
> "QTime":3
> },
> "overlay":{
> "znodeVersion":-1
> }
> }{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org