You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Gael Jourdan-Weil <ga...@kelkoogroup.com> on 2021/01/05 08:31:11 UTC
Sending compressed (gzip) UpdateRequest with SolrJ
Hello,
I was wondering if someone ever had the need to send compressed (gzip) update requests (adding/deleting documents), especially using SolrJ.
Somehow I expected it to be done by default, but didn't find any documentation about it and when looking at the code it seems there is no option to do it. Or is javabin compressed by default?
- https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/BinaryRequestWriter.java#L49
- https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/request/RequestWriter.java#L55 (if not using Javabin)
- https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java#L587
By the way, is there any documentation about javabin? I could only find one on the "old wiki".
Thanks,
Gaël
Re: Sending compressed (gzip) UpdateRequest with SolrJ
Posted by Walter Underwood <wu...@wunderwood.org>.
Years ago, working on the Ultraseek spider, we did a bunch of tests on compressed HTTP.
I expected it to be a big win, but the results were really inconclusive. Sometimes it was faster,
sometimes it was slower. We left it turned off.
It is an absolute win for serving already-compressed static content with Apache or whatever.
For dynamic content, it will increase some amount of delay as stuff is compressed before
sending. If the content already fits in one or two packets, it is just extra overhead. For really
large data, it helps with transmission time, but the processing time for large data probably
overwhelms the network time.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Jan 8, 2021, at 12:01 AM, Gael Jourdan-Weil <ga...@kelkoogroup.com> wrote:
>
> You're right Matthew.
>
> Jetty supports it for responses but for requests it doesn't seem to be the default.
> However I found a configuration not documented that needs to be set in the GzipHandler for it to work: inflateBufferSize.
>
> For SolrJ it still hacky to send gzip requests, maybe easier to use a regular http call..
>
> ---
>
> De : matthew sporleder <ms...@gmail.com>
> Envoyé : jeudi 7 janvier 2021 16:43
> À : solr-user@lucene.apache.org <so...@lucene.apache.org>
> Objet : Re: Sending compressed (gzip) UpdateRequest with SolrJ
>
> jetty supports http gzip and I've added it to solr before in my own
> installs (and submitted patches to do so by default to solr) but I
> don't know about the handling for solrj.
>
> IME compression helps a little, sometimes a lot, and never hurts.
> Even the admin interface benefits a lot from regular old http gzip
>
> On Thu, Jan 7, 2021 at 8:03 AM Gael Jourdan-Weil
> <ga...@kelkoogroup.com> wrote:
>>
>> Answering to myself on this one.
>>
>> Solr uses Jetty 9.x which does not support compressed requests by itself meaning, the application behind Jetty (that is Solr) has to decompress by itself which is not the case for now.
>> Thus even without using SolrJ, sending XML compressed in GZIP to Solr (with cURL for instance) is not possible for now.
>>
>> Seems quite surprising to me though.
>>
>> -----
>>
>> Hello,
>>
>> I was wondering if someone ever had the need to send compressed (gzip) update requests (adding/deleting documents), especially using SolrJ.
>>
>> Somehow I expected it to be done by default, but didn't find any documentation about it and when looking at the code it seems there is no option to do it. Or is javabin compressed by default?
>> - https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/BinaryRequestWriter.java#L49
>> - https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/request/RequestWriter.java#L55 (if not using Javabin)
>> - https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java#L587
>>
>> By the way, is there any documentation about javabin? I could only find one on the "old wiki".
>>
>> Thanks,
>> Gaël
RE: Sending compressed (gzip) UpdateRequest with SolrJ
Posted by Gael Jourdan-Weil <ga...@kelkoogroup.com>.
You're right Matthew.
Jetty supports it for responses but for requests it doesn't seem to be the default.
However I found a configuration not documented that needs to be set in the GzipHandler for it to work: inflateBufferSize.
For SolrJ it still hacky to send gzip requests, maybe easier to use a regular http call..
---
De : matthew sporleder <ms...@gmail.com>
Envoyé : jeudi 7 janvier 2021 16:43
À : solr-user@lucene.apache.org <so...@lucene.apache.org>
Objet : Re: Sending compressed (gzip) UpdateRequest with SolrJ
jetty supports http gzip and I've added it to solr before in my own
installs (and submitted patches to do so by default to solr) but I
don't know about the handling for solrj.
IME compression helps a little, sometimes a lot, and never hurts.
Even the admin interface benefits a lot from regular old http gzip
On Thu, Jan 7, 2021 at 8:03 AM Gael Jourdan-Weil
<ga...@kelkoogroup.com> wrote:
>
> Answering to myself on this one.
>
> Solr uses Jetty 9.x which does not support compressed requests by itself meaning, the application behind Jetty (that is Solr) has to decompress by itself which is not the case for now.
> Thus even without using SolrJ, sending XML compressed in GZIP to Solr (with cURL for instance) is not possible for now.
>
> Seems quite surprising to me though.
>
> -----
>
> Hello,
>
> I was wondering if someone ever had the need to send compressed (gzip) update requests (adding/deleting documents), especially using SolrJ.
>
> Somehow I expected it to be done by default, but didn't find any documentation about it and when looking at the code it seems there is no option to do it. Or is javabin compressed by default?
> - https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/BinaryRequestWriter.java#L49
> - https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/request/RequestWriter.java#L55 (if not using Javabin)
> - https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java#L587
>
> By the way, is there any documentation about javabin? I could only find one on the "old wiki".
>
> Thanks,
> Gaël
Re: Sending compressed (gzip) UpdateRequest with SolrJ
Posted by matthew sporleder <ms...@gmail.com>.
jetty supports http gzip and I've added it to solr before in my own
installs (and submitted patches to do so by default to solr) but I
don't know about the handling for solrj.
IME compression helps a little, sometimes a lot, and never hurts.
Even the admin interface benefits a lot from regular old http gzip
On Thu, Jan 7, 2021 at 8:03 AM Gael Jourdan-Weil
<ga...@kelkoogroup.com> wrote:
>
> Answering to myself on this one.
>
> Solr uses Jetty 9.x which does not support compressed requests by itself meaning, the application behind Jetty (that is Solr) has to decompress by itself which is not the case for now.
> Thus even without using SolrJ, sending XML compressed in GZIP to Solr (with cURL for instance) is not possible for now.
>
> Seems quite surprising to me though.
>
> -----
>
> Hello,
>
> I was wondering if someone ever had the need to send compressed (gzip) update requests (adding/deleting documents), especially using SolrJ.
>
> Somehow I expected it to be done by default, but didn't find any documentation about it and when looking at the code it seems there is no option to do it. Or is javabin compressed by default?
> - https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/BinaryRequestWriter.java#L49
> - https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/request/RequestWriter.java#L55 (if not using Javabin)
> - https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java#L587
>
> By the way, is there any documentation about javabin? I could only find one on the "old wiki".
>
> Thanks,
> Gaël
RE: Sending compressed (gzip) UpdateRequest with SolrJ
Posted by Gael Jourdan-Weil <ga...@kelkoogroup.com>.
Answering to myself on this one.
Solr uses Jetty 9.x which does not support compressed requests by itself meaning, the application behind Jetty (that is Solr) has to decompress by itself which is not the case for now.
Thus even without using SolrJ, sending XML compressed in GZIP to Solr (with cURL for instance) is not possible for now.
Seems quite surprising to me though.
-----
Hello,
I was wondering if someone ever had the need to send compressed (gzip) update requests (adding/deleting documents), especially using SolrJ.
Somehow I expected it to be done by default, but didn't find any documentation about it and when looking at the code it seems there is no option to do it. Or is javabin compressed by default?
- https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/BinaryRequestWriter.java#L49
- https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/request/RequestWriter.java#L55 (if not using Javabin)
- https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java#L587
By the way, is there any documentation about javabin? I could only find one on the "old wiki".
Thanks,
Gaël