Re: SolrJ connection leak with SolrCloud and Jetty Gzip compression enabled

[Samuel Garcia Martinez <sa...@inditex.com>]

I created two different issues: one for the Content-Type issue in the server and another one for the reliability issue in the SolrClient for unexpected/malformed responses.

ContentType issue: https://issues.apache.org/jira/browse/SOLR-14456
Client issue: https://issues.apache.org/jira/browse/SOLR-14457
________________________________
From: Jason Gerlowski <ge...@gmail.com>
Sent: Wednesday, April 22, 2020 4:43 PM
To: solr-user@lucene.apache.org <so...@lucene.apache.org>
Subject: Re: SolrJ connection leak with SolrCloud and Jetty Gzip compression enabled

Hi Samuel,

Thanks for the very detailed description of the problem here.  Very
thorough!  I don't think you're missing anything obvious, please file the
jira tickets if you haven't already.

Best,

Jason

On Mon, Apr 13, 2020 at 6:12 PM Samuel Garcia Martinez <
samuelgma@inditex.com> wrote:

> Reading again the last two paragraphs I realized that, those two
> specially, are very poorly worded (grammar 😓). I tried to rephrase them
> and correct some of the errors below.
>
> Here I can see three different problems:
>
> * HttpSolrCall should not use HttpServletResponse#setCharacterEncoding to
> set the Content-Encoding header. This is obviously a mistake.
> * HttpSolrClient, specifically the HttpClientUtil, should be modified to
> prevent that if the Content-Encoding header lies about the actual content,
> the connection is leaked forever. It should the exception though.
> * HttpSolrClient should allow clients to customize HttpClient's
> connectionRequestTimeout, preventing the application to be blocked forever
> waiting for a connection to be available. This way, the application could
> respond to requests that won’t use Solr instead of rejecting any incoming
> requests because all threads are blocked forever for a connection that
> won’t be available ever.
>
> I think the two first points are bugs that should be fixed.  The third one
> is a feature improvement to me.
>
> Unless I missed something, I'll file the two bugs and provide a patch for
> them. The same goes for the the feature improvement.
>
>
>
> Get Outlook for iOS<https://clicktime.symantec.com/3HCbuRyy1nsrbJk46YT1vS76H2?u=https%3A%2F%2Faka.ms%2Fo0ukef>
>
>
>
> En el caso de haber recibido este mensaje por error, le rogamos que nos lo
> comunique por esta misma vía, proceda a su eliminación y se abstenga de
> utilizarlo en modo alguno.
> If you receive this message by error, please notify the sender by return
> e-mail and delete it. Its use is forbidden.
>
>
>
> ________________________________
> From: Samuel Garcia Martinez <sa...@inditex.com>
> Sent: Monday, April 13, 2020 10:08:36 PM
> To: solr-user@lucene.apache.orG <so...@lucene.apache.orG>
> Subject: SolrJ connection leak with SolrCloud and Jetty Gzip compression
> enabled
>
> Hi!
>
> Today, I've seen a weird issue in production workloads when the gzip
> compression was enabled. After some minutes, the client app ran out of
> connections and stopped responding.
>
> The cluster setup is pretty simple:
> Solr version: 7.7.2
> Solr cloud enabled
> Cluster topology: 6 nodes, 1 single collection, 10 shards and 3 replicas.
> 1 HTTP LB using Round Robin over all nodes
> All cluster nodes have gzip enabled for all paths, all HTTP verbs and all
> MIME types.
> Solr client: HttpSolrClient targeting the HTTP LB
>
> Problem description: when the Solr node that receives the request has to
> forward the request to a Solr Node that actually can perform the query, the
> response headers are added incorrectly to the client response, causing the
> SolrJ client to fail and to never release the connection back to the pool.
>
> To simplify the case, let's try to start from the following repro scenario:
>
>   *   Start one node with cloud mode and port 8983
>   *   Create one single collection (1 shard, 1 replica)
>   *   Start another node with port 8984 and the previusly started zk (-z
> localhost:9983)
>   *   Start a java application and query the cluster using the node on
> port 8984 (the one that doesn't host the collection)
>
> So, the steps occur like:
>
>   *   The application queries node:8984 with compression enabled
> ("Accept-Encoding: gzip") and wt=javabin
>   *   Node:8984 can't perform the query and creates a http request behind
> the scenes to node:8983
>   *   Node:8983 returns a gzipped response with "Content-Encoding: gzip"
> and "Content-Type: application/octet-stream"
>   *   Node:8984 adds the "Content-Encoding: gzip" header as character
> stream to the response (it should be forwarded as "Content-Encoding"
> header, not character encoding)
>   *   HttpSolrClient receives a "Content-Type:
> application/octet-stream;charset=gzip", causing an exception.
>   *   HttpSolrClient tries to quietly close the connection, but since the
> stream is broken, the Utils.consumeFully fails to actually consume the
> entity (it throws another exception in GzipDecompressingEntity#getContent()
> with "not in GZIP format")
>
> The exception thrown by HttpSolrClient is:
> java.nio.charset.UnsupportedCharsetException: gzip
>                at java.nio.charset.Charset.forName(Charset.java:531)
>                at
> org.apache.http.entity.ContentType.create(ContentType.java:271)
>                at
> org.apache.http.entity.ContentType.create(ContentType.java:261)
>                at
> org.apache.http.entity.ContentType.parse(ContentType.java:319)
>                at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:591)
>                at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
>                at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
>                at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
>                at
> org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1015)
>                at
> org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1031)
>                at
> org.apache.solr.client.solrj.SolrClient$$FastClassBySpringCGLIB$$7fcf36a0.invoke(<generated>)
>                at
> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
>
> Here I can see three different problems:
>
>   *   HttpSolrCall should not use HttpServletResponse#setCharacterEncoding
> to set the Content-Encoding header. This is obviously a typo.
>   *   HttpSolrClient, specially the HttpClientUtil should be modified to
> prevent that if the Content-Encoding header lies about the actual content,
> there should be an exception, but shouldn't leak the connection forever.
>   *   HttpSolrClient should allow clients to customize HttpClient's
> connectionRequestTimeout, preventing the application to respond to any
> other incoming request because all requests that used could be forever
> blocked waiting for a free connection that will never be free.
>
> I think the two points are to bugs and the third one is a feature
> improvement. Unless I missed something, I'll file the two bugs and provide
> a patch for them. The same goes for the the feature improvement.
>
>
>
> En el caso de haber recibido este mensaje por error, le rogamos que nos lo
> comunique por esta misma v?a, proceda a su eliminaci?n y se abstenga de
> utilizarlo en modo alguno.
> If you receive this message by error, please notify the sender by return
> e-mail and delete it. Its use is forbidden.
>
>
>