Re: [DISCUSSION] Add DataStreamer's default per-node-batch-setting for PDS.

[Vladimir Steshin <vl...@gmail.com>]

     Hi, Stan.


     Thank you for the answer.

 >>>  "your data streamer queue size is something like"
You are right about writes queue on primary node. It has just some fixed 
size. But based on number of the CPUs. (x8). Even for my laptop I get 
16x8=128 batches. I wonder why so much by default for persistence.

 >>> "Can you check the heap dump in your tests to see what actually 
occupies most of the heap?"
The backup nodes collect `GridDhtAtomicSingleUpdateRequest`with key/data 
`byte[]`. That's where we don't wait for in this case.

     I thought we might slightly adjust the default setting at least to 
make simple test more reliable. As a user, I wouldn't like that I just 
take a tool/product just to try/research and it fails quick. But yes, 
user still has the related setting `perNodeParallelOperations()`

WDYT?

30.10.2022 21:24, Stanislav Lukyanov пишет:
> Hi Vladimir,
>
> I think this is potentially an issue but I don't think this is about PDS at all.
>
> The description is a bit vague, I have to say. AFAIU what you see is that when the caches are persistent the streamer writes data faster than the nodes (especially, backup nodes) process the writes.
> Therefore, the nodes accumulate the writes in the queues, the queues grow, and then you might go OOM.
>
> The solution to just have lesser queues when there is persistent (and therefore it's more likely the queues will reach the max size) is not the best one, in my opinion.
> If the default max queue size is too large, it should be less always, regardless of why the queues grow.
>
> Furthermore, I have a feeling that what gives you OOM isn't the data streamer queue... AFAIR your data streamer queue size is something like (entrySize * bufferSize * perNodeParallelOperations),
> which for 1 kb entries and 16 threads gives (1kb * 512 * 16 * 8) = 64mb which is usually peanuts for server Java.
>
> Can you check the heap dump in your tests to see what actually occupies most of the heap?
>
> Thanks,
> Stan
>
>> On 28 Oct 2022, at 11:54, Vladimir Steshin<vl...@gmail.com>  wrote:
>>
>>      Hi Folks,
>>
>>      I found that Datastreamer may consume heap or use increased heap amount when loading into a persistent cache.
>> This may happen with streamer's 'allowOverwite'==true and the cache is in PRIMARY_SYNC mode.
>>
>>      What I don't like here is that the case looks simple. Not the defaults, but user might meet the issue just in a trival test, trying/researching the streamer.
>>
>>      Streamer has related 'perNodeParallelOperations()' which helps. But addinional DFLT_PARALLEL_PERSISTENT_OPS_MULTIPLIER might be set for PDS.
>>
>>      My question are:
>> 1) Is it an issue at all? Need to fix? A minor?
>> 2) Should we bring additional default DFLT_PARALLEL_PERSISTENT_OPS_MULTIPLIER for PDS because it reduces heap consumption?
>> 3) Better solution is backpressure. But does it worth the case?
>>
>> Ticket:https://issues.apache.org/jira/browse/IGNITE-17735
>> PR:https://github.com/apache/ignite/pull/10343