RE: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

[Fikavec F <fi...@yandex.ru>]

Thanks to everyone for your help (especially Mikhail Khludnev and Michael
Gibney)!

I rewrote JSONResponseWriter using fasterxml jackson library
(com.fasterxml.jackson.core.JsonGenerator, I took the SmileResponseWriter code
as a basis) and added it to the collection as my own queryResponseWriter in
solrconfig.xml - in the test virtual infrastructure I got 4x+ acceleration in
json format without gzip (set http header 'Accept-Encoding: '), and about 2x
with gzip enabled (with set http header 'Accept-Encoding: gzip'). I think that
on real hardware, the acceleration may be a little more. CBOR from the jackson
library turned out to be also very fast and easy to add, it is also supported
in python by the cbor2 library and works very very fast, which is not to say
about smile.

Now we know for sure that Solr is capable of sending data in json format at
speeds of more than a few gigabits+, but the default JSONResponseWriter cannot
overcome 450 megabits - using the jackson library for serialization solves
this problem (<https://github.com/wizzardo/json-benchmarks)>.

Best Regards,

\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@solr.apache.org For additional commands,
e-mail: dev-help@solr.apache.org

Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

[Ishan Chattopadhyaya <ic...@gmail.com>]

Wow, that's absolutely fantastic. Would you like to contribute it upstream,
maybe a PR?

On Sun, 5 Mar, 2023, 9:21 pm Fikavec F, <fi...@yandex.ru> wrote:

> Thanks to everyone for your help (especially Mikhail Khludnev and Michael
> Gibney)!
>
>    I rewrote JSONResponseWriter using fasterxml jackson library
> (com.fasterxml.jackson.core.JsonGenerator, I took the SmileResponseWriter
> code as a basis) and added it to the collection as my own
> queryResponseWriter in solrconfig.xml - in the test virtual infrastructure
> I got 4x+ acceleration in json format without gzip (set http header
> 'Accept-Encoding: '), and about 2x with gzip enabled (with set http header
> 'Accept-Encoding: gzip'). I think that on real hardware, the acceleration
> may be a little more. CBOR from the jackson library turned out to be also
> very fast and easy to add, it is also supported in python by the cbor2
> library and works very very fast, which is not to say about smile.
>
>    Now we know for sure that Solr is capable of sending data in json
> format at speeds of more than a few gigabits+, but the default
> JSONResponseWriter cannot overcome 450 megabits - using the jackson library
> for serialization solves this problem (
> https://github.com/wizzardo/json-benchmarks).
>
> Best Regards,
> --------------------------------------------------------------------- To
> unsubscribe, e-mail: dev-unsubscribe@solr.apache.org For additional
> commands, e-mail: dev-help@solr.apache.org