Data Region Concurrency

[David Harvey <dh...@jobcase.com>]

We have a 2.4 ignite peristence cluster on AWS with 8 i3.xlarge nodes,
where the default memory region size is 160GB.    When loading a large
amount of data through a custom StreamReceiver, over time we see the
throughput decrease, as well as network and SSD I/O, and one node with
higher CPU usage than the others, but still only 33%.   Looking at that
node, I see (kill -3) that out of 27 sys-stripe* threads, 21 are blocked on
locks, 17 of which are memory segment locks.  The running threads have the
following call stacks:  acquirePage -> get/put ->getKey/Putkey .  The 2.4
documentation says I should have 32 (one per CPU) segments.    I wouldn't
expect such high contention if there was random distribution of load across
the segments.    I'm only using the default region.    Is there a
explaination for such a high contention level.

It looks as if I can set the number of segments higher by configuring the
data region

setConcurrencyLevel().   The 2.0 documentation for memory configuration
said the default was 4 * # CPUS, but the 2.4 Data Region code seems to
default to # CPUs, and we are only using default region.    What would be a
reasonable value?


It looks like the current data region settings get communicated to visor,
but I do not see a way to actually display them.   I'm I missing something?


If I change the xml configuration, it is unclear to me whether this will be
applied when I restart the node or is that setting only used when the data
region is originally created.   (As a general documentation thing, it would
be helpful to identify when settings will actually be applied. )

Thanks,
-DH

PS. Reading the code,   my first thought was that locks were configured as
fair, which ensures order but increases lock held time, but the default
seems to be unfair, which is good.   Fair  locks have the characteristic of
allowing high throughput until contended, then locking in low throughput
until the load completely stops.

Disclaimer

The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more visit the Mimecast website.

Re: Data Region Concurrency

[Dmitry Pavlov <dp...@gmail.com>]

Hi David,

Described behaviour (when too much time is spend on locks) was fixed in
Ignite 2.5 by 3 or 4 optimization changes.

IMO release will be published soon,so new Ignite vesion should get
performance boost in this case.

Yes, 'times 4' should be removed from the doc
https://apacheignite.readme.io/docs/memory-configuration#section-global-configuration-parameters


Sincerely,
Dmitriy Pavlov

вт, 29 мая 2018 г. в 5:42, Dave Harvey <dh...@jobcase.com>:

> It does appear that if the global concurrentLevel is not set, that it
> defaults to # CPUs not 4 * # CPUs as documented here
>
> https://apacheignite.readme.io/docs/memory-configuration#section-global-configuration-parameters
>
>
>    private long[] calculateFragmentSizes(int concLvl, long cacheSize, long
> chpBufSize) {
>         if (concLvl < 2)
>             concLvl = Runtime.getRuntime().availableProcessors();
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Data Region Concurrency

[Dave Harvey <dh...@jobcase.com>]

It does appear that if the global concurrentLevel is not set, that it
defaults to # CPUs not 4 * # CPUs as documented here
https://apacheignite.readme.io/docs/memory-configuration#section-global-configuration-parameters

    
   private long[] calculateFragmentSizes(int concLvl, long cacheSize, long
chpBufSize) {
        if (concLvl < 2)
            concLvl = Runtime.getRuntime().availableProcessors();



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/