[jira] [Updated] (IGNITE-20291) Possible memory leak in StreamerBuffer

["Aleksandr Polovtcev (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IGNITE-20291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aleksandr Polovtcev updated IGNITE-20291:
-----------------------------------------
    Description: 
I was running a stress test that simply inserted a lot of rows using {{RecordView#streamData}} method with the following options:

{code:java}
var options = DataStreamerOptions.builder()
        .perNodeParallelOperations(1)
        .batchSize(10_000)
        .autoFlushFrequency(100_000)
        .build();
{code}

After about 30 minutes I started getting Java OOM errors. Having inspected the heap dump, I found that all memory has been consumed by the Tuple instance that I have been inserting. The reason for that is the following:
# {{StreamerBuffer}} accumulates batches of data and then sends them to the server. If the server is slow for any reason, batch sending is placed into a "queue" by chaining {{flushFut}} one after another.
# {{StreamerSubscriber}} is the class that controls back-pressure. It uses two variables for this purpose: {{pendingItemCount}} which represents that amount of items, requested by the Subscriber, and {{inFlightItemCount}} which represents the amount of items being sent to the server. However, {{StreamerSubscriber}} does not know about the "queue" present in the {{StreamerBuffer}} and it increases the {{inFlightItemCount}} only when a particular flush future is being executed. This means that futures sitting in the "queue" do not contribute to the overall {{inFlightItemCount}} and {{StreamerSubscriber}} keeps requesting more items even though a lot of items are simply stuck in a different place.

> Possible memory leak in StreamerBuffer
> --------------------------------------
>
>                 Key: IGNITE-20291
>                 URL: https://issues.apache.org/jira/browse/IGNITE-20291
>             Project: Ignite
>          Issue Type: Bug
>            Reporter: Aleksandr Polovtcev
>            Assignee: Aleksandr Polovtcev
>            Priority: Major
>              Labels: ignite-3
>
> I was running a stress test that simply inserted a lot of rows using {{RecordView#streamData}} method with the following options:
> {code:java}
> var options = DataStreamerOptions.builder()
>         .perNodeParallelOperations(1)
>         .batchSize(10_000)
>         .autoFlushFrequency(100_000)
>         .build();
> {code}
> After about 30 minutes I started getting Java OOM errors. Having inspected the heap dump, I found that all memory has been consumed by the Tuple instance that I have been inserting. The reason for that is the following:
> # {{StreamerBuffer}} accumulates batches of data and then sends them to the server. If the server is slow for any reason, batch sending is placed into a "queue" by chaining {{flushFut}} one after another.
> # {{StreamerSubscriber}} is the class that controls back-pressure. It uses two variables for this purpose: {{pendingItemCount}} which represents that amount of items, requested by the Subscriber, and {{inFlightItemCount}} which represents the amount of items being sent to the server. However, {{StreamerSubscriber}} does not know about the "queue" present in the {{StreamerBuffer}} and it increases the {{inFlightItemCount}} only when a particular flush future is being executed. This means that futures sitting in the "queue" do not contribute to the overall {{inFlightItemCount}} and {{StreamerSubscriber}} keeps requesting more items even though a lot of items are simply stuck in a different place.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)