[jira] [Resolved] (HTTPCORE-726) SharedInputBuffer expands unbounded on large responses

["Oleg Kalnichevski (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HTTPCORE-726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Oleg Kalnichevski resolved HTTPCORE-726.
----------------------------------------
    Resolution: Fixed

[~johnlcox] I am glad to hear that. Many thanks for reporting the problem and providing a fix.

Oleg

> SharedInputBuffer expands unbounded on large responses
> ------------------------------------------------------
>
>                 Key: HTTPCORE-726
>                 URL: https://issues.apache.org/jira/browse/HTTPCORE-726
>             Project: HttpComponents HttpCore
>          Issue Type: Bug
>          Components: HttpCore
>    Affects Versions: 5.1.3
>         Environment: MacOS 12, Debian 9, Debian 10
>            Reporter: John Leacox
>            Priority: Major
>             Fix For: 5.1.5, 5.2-beta3
>
>         Attachments: SharedInputBuffer_Unbounded_Growth_Example.patch, SharedInputBuffer_Unbounded_Growth_Possible_Fix.patch
>
>
> Large responses can result in the {{SharedInputBuffer}} expanding unbounded, but with small intervals. Each expansion is the minimum 1024 byte aligned size bigger than the current buffer that is filling it beyond its current capacity. This leads to a ton of heap allocations and garbage collection each time is expands.
>  
> It looks like the capacity updates to {{CapacityChannel}} are reporting current capacity rather than deltas/increments, so that even when small reads occur, the whole capacity is reported again to the channel and the capacity window also grows unbounded.
>  
> I've attached a patch that includes some {{println}} statements for the capacity window size and for buffer being expanded along with the integration test {{testLargeResponseConsumer}} that shows the buffer and window growing quite large (the test passes, but the println statements show how large they are growing).
>  
> For a fix I was thinking of updating the capacity tracking to be similar to {{ReactiveDataConsumer}} in tracking the number of bytes read and then reporting that delta to {{CapacityChannel}}. This does seem to prevent the buffer from expanding to huge sizes, but it still expands more than I would expect because after each read from the {{SharedInputBuffer}}. This remaining expansion appears to be from {{CapacityWindow}} always setting the IO sessions with {{OP_READ}} even when the window is still negative after the update.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org