You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hc.apache.org by 石田隼也 <ju...@linecorp.com> on 2023/02/08 10:21:03 UTC
EntityUtils.toByteArray gets very slow in downloading large content with unknown content-length
Hi, our team uses `EntityUtils.toByteArray` method in `org.apache.httpcomponents:httpcore:4.4.15` to implement HTTP client to download a file.
We found that file download gets very slow when downloading a file of ~1.7 GB. Based on our investigation, the performance of copying the response to a byte array is significantly deteriorated in the case of large file download.
And, we also found that this perfomance problem is improved by using Java's `InputStream.readAllBytes` instead.
You can find the details of our investigation with benchmarks in the following repository.
* https://github.com/deftfitf/httpcomponent-entitiyutils-benchmarks
Based on this content, I believe that there is a room of improvement regarding the performance of `EntityUtils.toByteArray`.
Best regards,
Junya Ishida
Re: EntityUtils.toByteArray gets very slow in downloading large content with unknown content-length
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2023-02-08 at 19:21 +0900, 石田隼也 wrote:
> Hi, our team uses `EntityUtils.toByteArray` method in
> `org.apache.httpcomponents:httpcore:4.4.15` to implement HTTP client
> to download a file.
>
> We found that file download gets very slow when downloading a file of
> ~1.7 GB. Based on our investigation, the performance of copying the
> response to a byte array is significantly deteriorated in the case of
> large file download.
> And, we also found that this perfomance problem is improved by using
> Java's `InputStream.readAllBytes` instead.
>
> You can find the details of our investigation with benchmarks in the
> following repository.
> * https://github.com/deftfitf/httpcomponent-entitiyutils-benchmarks
>
> Based on this content, I believe that there is a room of improvement
> regarding the performance of `EntityUtils.toByteArray`.
>
>
> Best regards,
>
> Junya Ishida
One should _never_, _never_, ever consume HTTP message body as an array
of bytes. Never. Especially when message body can be greater than 1MB.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org