You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Antonio Conforti <an...@sia.eu> on 2019/02/22 14:37:41 UTC

Performance degradation in case of high volumes

Hi support,

I'm running a performance test writing 4000 entry per second on a cache:
1.	TRANSACTIONAL
2.	partitioned
3.	with backup 1 (and affinity with exclude neighbors enabled)
4.	write synchronization mode FULL_ASYNC
5.	indexed on key and value (and enabled to SQL inquiry)

Writes are performed by a client node using a data stream with StreamVisitor
and set autoFlushFrequency 1 sec.

We have configured:
1.	failureDetectionTimeout to 120000msec
2.	Data region (only 1):
a.	Persistence enabled
b.	max size 8 GB
c.	checkpointPageBufferSize 2 GB
3.	WAL mode LOG_ONLY
4.	disabled WAL archiving (WAL path and the WAL archive path to the same
value)
5.	Pages Writes Throttling enabled


After some hour submitting about 20 million entries without problems, the
client node starts to accuse delays: the queue from the client node Ignite
reads messages start to grow.

Verifying the logs of server and client node there isn’t any error message
but from the statistics of WAL  high FSYNC values are observed.

Could you help me to understand why inspite a constant rate and a constant
consumption of cpu of about 30% only after a certain amount of entry it
seems the server slow down in term of performance?

May be there is some param to tune that I missed?

Below the configuration used for the simulation:

Total server nodes  8 so distributed:
HOST1 with 4 nodes server and 1 client node on HDD disk
HOST2 with 4 nodes on HDD disk


Both hosts are machines with 16 cores of 256 GB of memory and HHD disk.

The DataStorageConfiguration for each server node is as follows:


<property name="dataStorageConfiguration">
                <bean
                       
class="org.apache.ignite.configuration.DataStorageConfiguration">
                        
                        <property name="writeThrottlingEnabled" value="true"
/>
                        <property name="defaultDataRegionConfiguration">
                                <bean
                                       
class="org.apache.ignite.configuration.DataRegionConfiguration">
                                        <property name="persistenceEnabled"
value="true" />
                                        <property name="maxSize" value="#{8L
* 1024 * 1024 * 1024}"/>   
                                        <property
name="checkpointPageBufferSize"
                                                value="#{2048L * 1024 *
1024}" />
                                </bean>
                        </property>

                        
                        
                        <property name="walMode" value="LOG_ONLY" />
                        <property name="walPath" value="wal/path" />
                        <property name="walArchivePath" value="wal/path" />
                </bean>
        </property>


JVM option used for start each server node:

-server -Xms4g -Xmx8g -XX:+AlwaysPreTouch -XX:+UseG1GC
-XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC



I report the WAL statistics from log of node 1 :

At Simulation start:
2019-02-22 10:19:44.195  INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=c115ead9-643d-45e5-be41-cd7ae5caac14, pages=11891,
markPos=FileWALPointer [idx=1, fileOff=36517886, len=79426],
walSegmentsCleared=0, walSegmentsCovered=[0], markDuration=34ms,
pagesWrite=87ms, fsync=1931ms, total=2052ms]
2019-02-22 10:22:44.742  INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=d40c9096-2e10-46ca-ae8e-2a39e242b768, pages=66732,
markPos=FileWALPointer [idx=7, fileOff=63806638, len=79426],
walSegmentsCleared=7, walSegmentsCovered=[1 - 6], markDuration=98ms,
pagesWrite=407ms, fsync=2085ms, total=2590ms]
2019-02-22 10:25:44.900  INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=5124f12f-6ed8-4ed3-9644-3c58957600ed, pages=70253,
markPos=FileWALPointer [idx=14, fileOff=47159207, len=79426],
walSegmentsCleared=6, walSegmentsCovered=[7 - 13], markDuration=98ms,
pagesWrite=402ms, fsync=2241ms, total=2741ms]
2019-02-22 10:28:47.866  INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=9094d36c-cd79-4d30-9b90-e48de78fa3e6, pages=72524,
markPos=FileWALPointer [idx=21, fileOff=39728290, len=79426],
walSegmentsCleared=8, walSegmentsCovered=[14 - 20], markDuration=83ms,
pagesWrite=365ms, fsync=5255ms, total=5703ms]
2019-02-22 10:31:53.635  INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=7132d53a-e2a6-4ac8-b1a8-2621cc39c82b, pages=77471,
markPos=FileWALPointer [idx=28, fileOff=64681287, len=79426],
walSegmentsCleared=7, walSegmentsCovered=[21 - 27], markDuration=494ms,
pagesWrite=748ms, fsync=10136ms, total=11472ms]


At end of simulation

2019-02-22 11:52:36.339  INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=dc8369b9-b100-4dd0-bfaf-9a5b13620072, pages=129942,
markPos=FileWALPointer [idx=309, fileOff=19048810, len=79426],
walSegmentsCleared=11, walSegmentsCovered=[298 - 308], markDuration=77ms,
pagesWrite=797ms, fsync=171049ms, total=171923ms]
2019-02-22 11:56:24.001  INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=1fb1ced8-a257-4c6f-be97-a1867ba6692e, pages=133096,
markPos=FileWALPointer [idx=320, fileOff=13420410, len=79426],
walSegmentsCleared=11, walSegmentsCovered=[309 - 319], markDuration=1707ms,
pagesWrite=1537ms, fsync=216332ms, total=219576ms]
2019-02-22 12:00:23.052  INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=0d48f6d5-7601-4839-b422-55b228978da5, pages=150587,
markPos=FileWALPointer [idx=332, fileOff=47800048, len=79426],
walSegmentsCleared=12, walSegmentsCovered=[320 - 331], markDuration=2275ms,
pagesWrite=752ms, fsync=236023ms, total=239051ms]
2019-02-22 12:04:05.562  INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=b23526c4-5121-48a9-9902-522a5ffb3a28, pages=155805,
markPos=FileWALPointer [idx=345, fileOff=40020477, len=79426],
walSegmentsCleared=13, walSegmentsCovered=[332 - 344], markDuration=525ms,
pagesWrite=1324ms, fsync=220654ms, total=222504ms]
2019-02-22 12:07:54.005  INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=1bb2a0c0-7a89-47f2-af9f-90e90d44c14b, pages=149055,
markPos=FileWALPointer [idx=357, fileOff=51666923, len=79426],
walSegmentsCleared=12, walSegmentsCovered=[345 - 356], markDuration=995ms,
pagesWrite=1559ms, fsync=225888ms, total=228442ms]
2019-02-22 12:11:49.962  INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=19f14a04-842f-409e-9c6b-afb59193e419, pages=153022,
markPos=FileWALPointer [idx=370, fileOff=16234647, len=79426],
walSegmentsCleared=13, walSegmentsCovered=[357 - 369], markDuration=1773ms,
pagesWrite=1044ms, fsync=233139ms, total=235957ms]
2019-02-22 12:15:59.332  INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=c1316fb7-1ecc-4358-bf90-a87772969c03, pages=159668,
markPos=FileWALPointer [idx=383, fileOff=21979375, len=79426],
walSegmentsCleared=13, walSegmentsCovered=[370 - 382], markDuration=1249ms,
pagesWrite=1693ms, fsync=246428ms, total=249370ms]
2019-02-22 12:20:05.814  INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=a9e394f2-0011-4f89-8b5a-a6ed77774103, pages=156891,
markPos=FileWALPointer [idx=396, fileOff=3956799, len=79426],
walSegmentsCleared=13, walSegmentsCovered=[383 - 395], markDuration=1030ms,
pagesWrite=1275ms, fsync=244176ms, total=246482ms]
2019-02-22 12:24:40.217  INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=0bc71cc5-97c4-4f7d-95c2-430f379eeeb0, pages=148039,
markPos=FileWALPointer [idx=407, fileOff=57767331, len=79426],
walSegmentsCleared=11, walSegmentsCovered=[396 - 406], markDuration=323ms,
pagesWrite=1620ms, fsync=272460ms, total=274403ms]


Thanks in advance.

Antonio




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Performance degradation in case of high volumes

Posted by Justin Ji <bi...@foxmail.com>.
Ilya - 

I use ignite 2.6.0, does not have IGNITE_MAX_INDEX_PAYLOAD_SIZE system
property.
But our index field has a fixed length:25 characters, so where can I find
the algorithm to calculate the 'index inline size'.

Looking forward to your reply.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Performance degradation in case of high volumes

Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!

I'm afraid that at this point even indexes do not fit in RAM so every
insert needs to get index pages from disk, change them and put them back to
make room for newer ones. Please consider having larger DataRegions.

Regards,
-- 
Ilya Kasnacheev


ср, 6 мар. 2019 г. в 16:26, Antonio Conforti <an...@sia.eu>:

> Hello Ilya,
>
>
> thanks for reply.
>
> For answer to your questions today i runned again a test that i stopped at
> about 22 millions of entries:
>
> a) the approximate hardware and data region config used in the test:
>
> 2 hosts each with:
>
> Processor total  : 2
>
> Processor: 0
>         Name         : Intel Xeon
>         Speed        : 3300 MHz
>         Bus          : 100 MHz
>         Core         : 8
>         Thread       : 16
>         Socket       : 1
>         Level2 Cache : 2048 KBytes
>         Level3 Cache : 25600 KBytes
>
> Processor: 1
>         Name         : Intel Xeon
>         Stepping     : 4
>         Speed        : 3300 MHz
>         Bus          : 100 MHz
>         Core         : 8
>         Thread       : 16
>         Socket       : 2
>         Level2 Cache : 2048 KBytes
>         Level3 Cache : 25600 KBytes
>
> MEMORY:
> Memory installed : 196608 MBytes
>
> # getconf PAGESIZE: 4096
>
> vm.swappiness = 5
>
>
>
>
>
> Disk:
>
> Cache Board Present: True
> Cache Status: OK
> Cache Ratio: 25% Read / 75% Write
> Drive Write Cache: Disabled
> Total Cache Size: 512 MB
> Total Cache Memory Available: 304 MB
>
> Logical Drive: 1
>                 Size: 279.4 GB
>                 Fault Tolerance: RAID 1
>                 Heads: 255
>                 Sectors Per Track: 32
>                 Cylinders: 65535
>                 Strip Size: 256 KB
>                 Full Stripe Size: 256 KB
>                 Status: OK
>                 Caching:  Enabled
>                 Rotational Speed: 15000
>                 PHY Transfer Rate: 6.0Gbps
>                 PHY Transfer Rate: 6.0Gbps
>                 type ext4
>                 Block size:  4096
>
>
> From the attached xml config the default data storage configuration is:
>
>
> datafeed-persistent-store.xml
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t2315/datafeed-persistent-store.xml>
>
>
> <property name="maxSize" value="#{8L * 1024 * 1024 * 1024}"/>
>
> <property name="checkpointPageBufferSize" value="#{2048L * 1024 * 1024}" />
>
>
>  About the memory below the top command for server and client nodes before
> start test:
>
>
>
>
> HOST_1
>
>
>
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t2315/HOST1_top_preTest.png>
>
>
> HOST_2
>
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t2315/HOST2_top_preTest.png>
>
>
> and below the top command for server and client nodes when test is stopped:
>
> HOST_1
>
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t2315/HOST1_top_postTest.png>
>
>
> HOST_2
>
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t2315/HOST2_top_postTest.png>
>
>
>
> b) approximate size of data is 656 bytes
> c) The size of db/ directory at about  22 millions is
>
> For HOST_1:
> 6.7G    Work_1/
> 5.3G    Work_3/
> 6.3G    Work_5/
> 7.2G    Work_7/
>
>
>
> For HOST_2:
> 6.5G    Work_2
> 6.5G    Work_4
> 6.6G    Work_6
> 5.9G    Work_8
>
> Thanks,
>
> Antonio
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Performance degradation in case of high volumes

Posted by Antonio Conforti <an...@sia.eu>.
Hello Ilya,


thanks for reply.

For answer to your questions today i runned again a test that i stopped at
about 22 millions of entries:

a) the approximate hardware and data region config used in the test:

2 hosts each with:

Processor total  : 2

Processor: 0
        Name         : Intel Xeon
        Speed        : 3300 MHz
        Bus          : 100 MHz
        Core         : 8
        Thread       : 16
        Socket       : 1
        Level2 Cache : 2048 KBytes
        Level3 Cache : 25600 KBytes

Processor: 1
        Name         : Intel Xeon
        Stepping     : 4
        Speed        : 3300 MHz
        Bus          : 100 MHz
        Core         : 8
        Thread       : 16
        Socket       : 2
        Level2 Cache : 2048 KBytes
        Level3 Cache : 25600 KBytes

MEMORY:
Memory installed : 196608 MBytes

# getconf PAGESIZE: 4096

vm.swappiness = 5





Disk:

Cache Board Present: True
Cache Status: OK
Cache Ratio: 25% Read / 75% Write
Drive Write Cache: Disabled
Total Cache Size: 512 MB
Total Cache Memory Available: 304 MB

Logical Drive: 1
                Size: 279.4 GB
                Fault Tolerance: RAID 1
                Heads: 255
                Sectors Per Track: 32
                Cylinders: 65535
                Strip Size: 256 KB
                Full Stripe Size: 256 KB
                Status: OK
                Caching:  Enabled
                Rotational Speed: 15000
                PHY Transfer Rate: 6.0Gbps
                PHY Transfer Rate: 6.0Gbps
                type ext4
                Block size:  4096


From the attached xml config the default data storage configuration is:


datafeed-persistent-store.xml
<http://apache-ignite-users.70518.x6.nabble.com/file/t2315/datafeed-persistent-store.xml>  

<property name="maxSize" value="#{8L * 1024 * 1024 * 1024}"/>                                           
<property name="checkpointPageBufferSize" value="#{2048L * 1024 * 1024}" />


 About the memory below the top command for server and client nodes before
start test:




HOST_1



<http://apache-ignite-users.70518.x6.nabble.com/file/t2315/HOST1_top_preTest.png> 

HOST_2

<http://apache-ignite-users.70518.x6.nabble.com/file/t2315/HOST2_top_preTest.png> 

and below the top command for server and client nodes when test is stopped:

HOST_1

<http://apache-ignite-users.70518.x6.nabble.com/file/t2315/HOST1_top_postTest.png> 

HOST_2

<http://apache-ignite-users.70518.x6.nabble.com/file/t2315/HOST2_top_postTest.png> 


b) approximate size of data is 656 bytes
c) The size of db/ directory at about  22 millions is

For HOST_1:
6.7G    Work_1/
5.3G    Work_3/
6.3G    Work_5/
7.2G    Work_7/



For HOST_2:
6.5G    Work_2
6.5G    Work_4
6.6G    Work_6
5.9G    Work_8

Thanks,

Antonio





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Performance degradation in case of high volumes

Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!

Can you please share a) approximate hardware and data region config that
you are using, and b) approximate size of your data.

How large is 18 million entries? What's the size of db/ directory at this
point?

Regards,
-- 
Ilya Kasnacheev


пн, 4 мар. 2019 г. в 17:36, Antonio Conforti <an...@sia.eu>:

> Hello Ilya,
>
> not running out of checkpoint buffers is certainly good news..
>
> I run today another test to collect the information you asked for, with the
> same configuration of the test executed in date 2019/03/01 and previously
> discussed in this thread.
> Below the details of entries per node :
>
> Cache 'DF_CMF_QUOTE(@c0)':
> +------------------------------------------------------------------+
> | Name(@)                         | DF_CMF_QUOTE(@c0)              |
> | Total entries (Heap / Off-heap) | 24050018 (0 / 24050018)        |
> | Nodes                           | 8                              |
> | Total size Min/Avg/Max          | 2419854 / 3006252.25 / 3426152 |
> |   Heap size Min/Avg/Max         | 0 / 0.00 / 0                   |
> |   Off-heap size Min/Avg/Max     | 2419854 / 3006252.25 / 3426152 |
> +------------------------------------------------------------------+
>
> Nodes for: DF_CMF_QUOTE(@c0)
>
> +==============================================================================================================================+
> |        Node ID8(@), IP        | CPUs | Heap Used | CPU Load |   Up Time
>
> |      Size (Primary / Backup)      | Hi/Mi/Rd/Wr |
>
> +==============================================================================================================================+
> | 269DFE65(@n7),        HOST_1  | 16   | 2.67 %    | 0.03 %   |
> 02:25:29.411
> | Total: 3147250 (3147250 / 0)      | Hi: 0       |
> |                               |      |           |          |
>
> |   Heap: 0 (0 / <n/a>)             | Mi: 0       |
> |                               |      |           |          |
>
> |   Off-Heap: 3147250 (3147250 / 0) | Rd: 0       |
> |                               |      |           |          |
>
> |   Off-Heap Memory: <n/a>          | Wr: 0       |
>
> +-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
> | FF347987(@n6),        HOST_1  | 16   | 23.93 %   | 0.03 %   |
> 02:25:17.106
> | Total: 2820298 (2820298 / 0)      | Hi: 0       |
> |                               |      |           |          |
>
> |   Heap: 0 (0 / <n/a>)             | Mi: 0       |
> |                               |      |           |          |
>
> |   Off-Heap: 2820298 (2820298 / 0) | Rd: 0       |
> |                               |      |           |          |
>
> |   Off-Heap Memory: <n/a>          | Wr: 0       |
>
> +-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
> | 5DF4A6EE(@n4),        HOST_1  | 16   | 12.12 %   | 0.00 %   |
> 02:25:34.673
> | Total: 3077901 (3077901 / 0)      | Hi: 0       |
> |                               |      |           |          |
>
> |   Heap: 0 (0 / <n/a>)             | Mi: 0       |
> |                               |      |           |          |
>
> |   Off-Heap: 3077901 (3077901 / 0) | Rd: 0       |
> |                               |      |           |          |
>
> |   Off-Heap Memory: <n/a>          | Wr: 0       |
>
> +-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
> | 51133869(@n5),        HOST_1  | 16   | 2.44 %    | 0.00 %   |
> 02:25:23.450
> | Total: 3146224 (3146224 / 0)      | Hi: 0       |
> |                               |      |           |          |
>
> |   Heap: 0 (0 / <n/a>)             | Mi: 0       |
> |                               |      |           |          |
>
> |   Off-Heap: 3146224 (3146224 / 0) | Rd: 0       |
> |                               |      |           |          |
>
> |   Off-Heap Memory: <n/a>          | Wr: 0       |
>
> +-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
> | E4CC3158(@n0),        HOST_2  | 16   | 5.15 %    | 0.00 %   |
> 02:26:51.374
> | Total: 3073950 (3073950 / 0)      | Hi: 0       |
> |                               |      |           |          |
>
> |   Heap: 0 (0 / <n/a>)             | Mi: 0       |
> |                               |      |           |          |
>
> |   Off-Heap: 3073950 (3073950 / 0) | Rd: 0       |
> |                               |      |           |          |
>
> |   Off-Heap Memory: <n/a>          | Wr: 0       |
>
> +-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
> | EB897A74(@n2),        HOST_2  | 16   | 8.85 %    | 0.03 %   |
> 02:26:36.007
> | Total: 2938389 (2938389 / 0)      | Hi: 0       |
> |                               |      |           |          |
>
> |   Heap: 0 (0 / <n/a>)             | Mi: 0       |
> |                               |      |           |          |
>
> |   Off-Heap: 2938389 (2938389 / 0) | Rd: 0       |
> |                               |      |           |          |
>
> |   Off-Heap Memory: <n/a>          | Wr: 0       |
>
> +-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
> | 38404C41(@n3),        HOST_2  | 16   | 5.49 %    | 0.03 %   |
> 02:26:29.103
> | Total: 3426152 (3426152 / 0)      | Hi: 0       |
> |                               |      |           |          |
>
> |   Heap: 0 (0 / <n/a>)             | Mi: 0       |
> |                               |      |           |          |
>
> |   Off-Heap: 3426152 (3426152 / 0) | Rd: 0       |
> |                               |      |           |          |
>
> |   Off-Heap Memory: <n/a>          | Wr: 0       |
>
> +-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
> | 570D0880(@n1),        HOST_2  | 16   | 26.10 %   | 0.03 %   |
> 02:26:41.175
> | Total: 2419854 (2419854 / 0)      | Hi: 0       |
> |                               |      |           |          |
>
> |   Heap: 0 (0 / <n/a>)             | Mi: 0       |
> |                               |      |           |          |
>
> |   Off-Heap: 2419854 (2419854 / 0) | Rd: 0       |
> |                               |      |           |          |
>
> |   Off-Heap Memory: <n/a>          | Wr: 0       |
>
> +------------------------------------------------------------------------------------------------------------------------------+
>
> Some comments about the test:
>
> 1) When I start the test with the cache empty, I begin to observe the
> degradation of performance after about 1 hour and 15 minutes and,
> particularly, when I'm around a total of 18 million entries.
>  Before that number of entries everything seems to be fine.
>
>    Below the statistics at the end of the checkpoint read in log file for
> node 7:
>
> Run A)  for CONSISTENDID 7
> Stop time checkPoint    Num Pages        Elapsed Checkpoint ms
> 09:25:49.016                                      18353
> 449
> 09:28:49.941                                      59593
> 1368
> 09:31:50.057                                      62797
> 1480
> 09:34:50.234                                      65509
> 1655
> 09:37:50.489                                      69339
> 1902
> 09:40:56.005                                      72438
> 7410
> 09:44:03.367                                      74923
> 14767
> 09:47:07.403                                      78554
> 18799
> 09:50:13.981                                      81913
> 25372
> 09:53:24.503                                      85730
> 35889
> 09:56:27.091                                      88078
> 38468
> 09:59:31.535                                      90275
> 42913
> 10:02:35.038                                      92802
> 46409
> 10:05:40.214                                      95503
> 51579
> 10:08:49.639                                      97856
> 60996
> 10:11:58.841                                      101746
> 70203
> 10:15:02.559                                      104501
> 73913
> 10:18:14.915                                      106753
> 86266
> 10:21:17.593                                      108685
> 88942
> 10:24:25.780                                      111062
> 97126
> 10:27:33.036                                      112904
> 104376
> 10:30:35.121                                      113809
> 106461
> 10:33:40.040                                      115920
> 111374
> 10:39:46.645                                      118431
> 117973
> 10:42:54.245                                      120306
> 125568
> 10:45:56.420                                      121766
> 127737
> 10:49:02.531                                      123532
> 133845
> 10:52:04.470                                      125685
> 135785
> 10:55:12.588                                      127601
> 143895
> 10:58:16.771                                      129164
> 148075
> 11:00:59.904                                      102594
> 131206
>
>
>
> 2) If I stop the test when I begin to observe the degradation of
> performance
> described in point 1) at about 11:00 AM and I wait for a while, precisely
> since in the log of my servers nodes I read "Skipping checkpoint (no pages
> were modified)" in order to be sure that no pending entries are in
> processing and then I run again the test submitting 4000 entry per second,
> I
> observe in a short time a degradation of the performances.
>
> Below the statistic at the end of the checkpoint read in log file fornode
> 7:
>
> Run B)  for CONSISTENDID 7
> Stop time checkPoint    Num Pages        Elapsed Checkpoint ms
> 11:23:09.120                                      24743
> 20386
> 11:28:58.436                                      131858
> 189695
> 11:32:45.824                                      137487
> 227388
> 11:33:53.565                                      53774
> 67741
>
>
> Please note also as highlighted above in table of point 1) the progressive
> grow of pages and consequent elapsed in the execution times of checkpoints.
> A note also inherent the pages managed by checkpoint in run b) described in
> point 2) compared to pages managed by checkpoint of run A) described in
> point 1):
> the number of pages managed at the start of run B) are quite the same as
> those at the end of run A) and not, as I expected, similar to those at the
> start of the initial run A).
>
> The observations above might suggest to tune some other configuration
> parameter?
>
> What may be, in your opinion,  the cause of this performance degradation
> when I reach about 18 milions of entries?
> What system resource may be exhausted ?
>
> Can you also tell me in what log do you observe the different
> configurations
> ? The configuration file used for all nodes seems to me the same but
> probably I'm missing something ...
>
> In attach the logs about the run test 2019/03/01
> log_ignite_190304_HOST1.gz
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t2315/log_ignite_190304_HOST1.gz>
>
> log_ignite_190304_HOST2.gz
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t2315/log_ignite_190304_HOST2.gz>
>
>
> Thanks,
> Antonio
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Performance degradation in case of high volumes

Posted by Antonio Conforti <an...@sia.eu>.
Hello Ilya,

not running out of checkpoint buffers is certainly good news..

I run today another test to collect the information you asked for, with the
same configuration of the test executed in date 2019/03/01 and previously
discussed in this thread.
Below the details of entries per node : 

Cache 'DF_CMF_QUOTE(@c0)':
+------------------------------------------------------------------+
| Name(@)                         | DF_CMF_QUOTE(@c0)              |
| Total entries (Heap / Off-heap) | 24050018 (0 / 24050018)        |
| Nodes                           | 8                              |
| Total size Min/Avg/Max          | 2419854 / 3006252.25 / 3426152 |
|   Heap size Min/Avg/Max         | 0 / 0.00 / 0                   |
|   Off-heap size Min/Avg/Max     | 2419854 / 3006252.25 / 3426152 |
+------------------------------------------------------------------+

Nodes for: DF_CMF_QUOTE(@c0)
+==============================================================================================================================+
|        Node ID8(@), IP        | CPUs | Heap Used | CPU Load |   Up Time   
|      Size (Primary / Backup)      | Hi/Mi/Rd/Wr |
+==============================================================================================================================+
| 269DFE65(@n7),        HOST_1  | 16   | 2.67 %    | 0.03 %   | 02:25:29.411
| Total: 3147250 (3147250 / 0)      | Hi: 0       |
|                               |      |           |          |             
|   Heap: 0 (0 / <n/a>)             | Mi: 0       |
|                               |      |           |          |             
|   Off-Heap: 3147250 (3147250 / 0) | Rd: 0       |
|                               |      |           |          |             
|   Off-Heap Memory: <n/a>          | Wr: 0       |
+-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
| FF347987(@n6),        HOST_1  | 16   | 23.93 %   | 0.03 %   | 02:25:17.106
| Total: 2820298 (2820298 / 0)      | Hi: 0       |
|                               |      |           |          |             
|   Heap: 0 (0 / <n/a>)             | Mi: 0       |
|                               |      |           |          |             
|   Off-Heap: 2820298 (2820298 / 0) | Rd: 0       |
|                               |      |           |          |             
|   Off-Heap Memory: <n/a>          | Wr: 0       |
+-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
| 5DF4A6EE(@n4),        HOST_1  | 16   | 12.12 %   | 0.00 %   | 02:25:34.673
| Total: 3077901 (3077901 / 0)      | Hi: 0       |
|                               |      |           |          |             
|   Heap: 0 (0 / <n/a>)             | Mi: 0       |
|                               |      |           |          |             
|   Off-Heap: 3077901 (3077901 / 0) | Rd: 0       |
|                               |      |           |          |             
|   Off-Heap Memory: <n/a>          | Wr: 0       |
+-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
| 51133869(@n5),        HOST_1  | 16   | 2.44 %    | 0.00 %   | 02:25:23.450
| Total: 3146224 (3146224 / 0)      | Hi: 0       |
|                               |      |           |          |             
|   Heap: 0 (0 / <n/a>)             | Mi: 0       |
|                               |      |           |          |             
|   Off-Heap: 3146224 (3146224 / 0) | Rd: 0       |
|                               |      |           |          |             
|   Off-Heap Memory: <n/a>          | Wr: 0       |
+-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
| E4CC3158(@n0),        HOST_2  | 16   | 5.15 %    | 0.00 %   | 02:26:51.374
| Total: 3073950 (3073950 / 0)      | Hi: 0       |
|                               |      |           |          |             
|   Heap: 0 (0 / <n/a>)             | Mi: 0       |
|                               |      |           |          |             
|   Off-Heap: 3073950 (3073950 / 0) | Rd: 0       |
|                               |      |           |          |             
|   Off-Heap Memory: <n/a>          | Wr: 0       |
+-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
| EB897A74(@n2),        HOST_2  | 16   | 8.85 %    | 0.03 %   | 02:26:36.007
| Total: 2938389 (2938389 / 0)      | Hi: 0       |
|                               |      |           |          |             
|   Heap: 0 (0 / <n/a>)             | Mi: 0       |
|                               |      |           |          |             
|   Off-Heap: 2938389 (2938389 / 0) | Rd: 0       |
|                               |      |           |          |             
|   Off-Heap Memory: <n/a>          | Wr: 0       |
+-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
| 38404C41(@n3),        HOST_2  | 16   | 5.49 %    | 0.03 %   | 02:26:29.103
| Total: 3426152 (3426152 / 0)      | Hi: 0       |
|                               |      |           |          |             
|   Heap: 0 (0 / <n/a>)             | Mi: 0       |
|                               |      |           |          |             
|   Off-Heap: 3426152 (3426152 / 0) | Rd: 0       |
|                               |      |           |          |             
|   Off-Heap Memory: <n/a>          | Wr: 0       |
+-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
| 570D0880(@n1),        HOST_2  | 16   | 26.10 %   | 0.03 %   | 02:26:41.175
| Total: 2419854 (2419854 / 0)      | Hi: 0       |
|                               |      |           |          |             
|   Heap: 0 (0 / <n/a>)             | Mi: 0       |
|                               |      |           |          |             
|   Off-Heap: 2419854 (2419854 / 0) | Rd: 0       |
|                               |      |           |          |             
|   Off-Heap Memory: <n/a>          | Wr: 0       |
+------------------------------------------------------------------------------------------------------------------------------+

Some comments about the test:

1) When I start the test with the cache empty, I begin to observe the
degradation of performance after about 1 hour and 15 minutes and,
particularly, when I'm around a total of 18 million entries.
 Before that number of entries everything seems to be fine.

   Below the statistics at the end of the checkpoint read in log file for
node 7:
   
Run A)  for CONSISTENDID 7
Stop time checkPoint    Num Pages        Elapsed Checkpoint ms
09:25:49.016                                      18353                   
449
09:28:49.941                                      59593                   
1368
09:31:50.057                                      62797                   
1480
09:34:50.234                                      65509                   
1655
09:37:50.489                                      69339                   
1902
09:40:56.005                                      72438                   
7410
09:44:03.367                                      74923                   
14767
09:47:07.403                                      78554                   
18799
09:50:13.981                                      81913                   
25372
09:53:24.503                                      85730                   
35889
09:56:27.091                                      88078                   
38468
09:59:31.535                                      90275                   
42913
10:02:35.038                                      92802                   
46409
10:05:40.214                                      95503                   
51579
10:08:49.639                                      97856                   
60996
10:11:58.841                                      101746                
70203
10:15:02.559                                      104501                
73913
10:18:14.915                                      106753                
86266
10:21:17.593                                      108685                
88942
10:24:25.780                                      111062                
97126
10:27:33.036                                      112904                
104376
10:30:35.121                                      113809                
106461
10:33:40.040                                      115920                
111374
10:39:46.645                                      118431                
117973
10:42:54.245                                      120306                
125568
10:45:56.420                                      121766                
127737
10:49:02.531                                      123532                
133845
10:52:04.470                                      125685                
135785
10:55:12.588                                      127601                
143895
10:58:16.771                                      129164                
148075
11:00:59.904                                      102594                
131206



2) If I stop the test when I begin to observe the degradation of performance
described in point 1) at about 11:00 AM and I wait for a while, precisely
since in the log of my servers nodes I read "Skipping checkpoint (no pages
were modified)" in order to be sure that no pending entries are in
processing and then I run again the test submitting 4000 entry per second, I
observe in a short time a degradation of the performances.

Below the statistic at the end of the checkpoint read in log file fornode 7:

Run B)  for CONSISTENDID 7                      
Stop time checkPoint    Num Pages        Elapsed Checkpoint ms
11:23:09.120                                      24743                   
20386
11:28:58.436                                      131858                
189695
11:32:45.824                                      137487                
227388
11:33:53.565                                      53774                   
67741


Please note also as highlighted above in table of point 1) the progressive
grow of pages and consequent elapsed in the execution times of checkpoints.
A note also inherent the pages managed by checkpoint in run b) described in
point 2) compared to pages managed by checkpoint of run A) described in
point 1):
the number of pages managed at the start of run B) are quite the same as
those at the end of run A) and not, as I expected, similar to those at the
start of the initial run A). 

The observations above might suggest to tune some other configuration
parameter?

What may be, in your opinion,  the cause of this performance degradation
when I reach about 18 milions of entries? 
What system resource may be exhausted ?

Can you also tell me in what log do you observe the different configurations
? The configuration file used for all nodes seems to me the same but
probably I'm missing something ...

In attach the logs about the run test 2019/03/01
log_ignite_190304_HOST1.gz
<http://apache-ignite-users.70518.x6.nabble.com/file/t2315/log_ignite_190304_HOST1.gz>  
log_ignite_190304_HOST2.gz
<http://apache-ignite-users.70518.x6.nabble.com/file/t2315/log_ignite_190304_HOST2.gz>  

Thanks,
Antonio




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Performance degradation in case of high volumes

Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello Antonio!

I canonly observe 'timeout' checkpoints which is good news - you are not
running out of checkpoint buffer.

Otherwise, maybe you are hitting actual performance limit, i.e., your
system is saturated for good at this point. What is total amount of data
per node at this time? What's the size of your cache entry?

I have also noted that your 2 nodes have wildly different config. The first
one has around 39G of data region, the second one has 8G.

Regards,
-- 
Ilya Kasnacheev


пт, 1 мар. 2019 г. в 17:19, Antonio Conforti <an...@sia.eu>:

> Hello Ilya.
>
> I ran again the test from scratch with fixed rate at 4000 msg/sec. with the
> environment variable IGNITE_MAX_INDEX_PAYLOAD_SIZE=66 and Cache:
>
> 1) PARTITIONED
> 2) TRANSACTIONAL
> 3) persistence enabled
> 4) backup=0
> 5) indexes on key and value
> 6) Data region 8 GB
> 7) Checkpoint buffer size 2 GB
> 8) WAL mode LOG_ONLY
> 9) WAL archive disabled
> 10)Pages Writes Throttling enabled)
>
> During the test I could observe the total checkpoint elapsed grew up and
> the
> platform processed all the messages without queueing. The throttling  was
> logged in every checkpoint.
> After about 18 milions of entries  the performance slowdown and queues
> formed. The total checkpoint elapsed exceeded the checkpoint timeout
> (default).
> A lot of "Critical system error detected" were logged and the platform
> never
> recovered the performance.
> I observed also the data region was filled after the perfomance slowdown
> and
> the checkpoint elapsed exceeded the timeout.
>
> In attach the logs and configuration files.
>
> log_ignite_190301_HOST1.gz
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t2315/log_ignite_190301_HOST1.gz>
>
> log_ignite_190301_HOST2.gz
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t2315/log_ignite_190301_HOST2.gz>
>
>
> Waiting your suggestions let me know if may be usueful a reproducer project
> for the slow down of performance that i observed
>
> Thanks.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Performance degradation in case of high volumes

Posted by Antonio Conforti <an...@sia.eu>.
Hello Ilya.

I ran again the test from scratch with fixed rate at 4000 msg/sec. with the
environment variable IGNITE_MAX_INDEX_PAYLOAD_SIZE=66 and Cache:

1) PARTITIONED
2) TRANSACTIONAL
3) persistence enabled
4) backup=0
5) indexes on key and value
6) Data region 8 GB
7) Checkpoint buffer size 2 GB
8) WAL mode LOG_ONLY
9) WAL archive disabled
10)Pages Writes Throttling enabled)

During the test I could observe the total checkpoint elapsed grew up and the
platform processed all the messages without queueing. The throttling  was
logged in every checkpoint.
After about 18 milions of entries  the performance slowdown and queues
formed. The total checkpoint elapsed exceeded the checkpoint timeout
(default).
A lot of "Critical system error detected" were logged and the platform never
recovered the performance.
I observed also the data region was filled after the perfomance slowdown and
the checkpoint elapsed exceeded the timeout.

In attach the logs and configuration files.

log_ignite_190301_HOST1.gz
<http://apache-ignite-users.70518.x6.nabble.com/file/t2315/log_ignite_190301_HOST1.gz>  
log_ignite_190301_HOST2.gz
<http://apache-ignite-users.70518.x6.nabble.com/file/t2315/log_ignite_190301_HOST2.gz>  

Waiting your suggestions let me know if may be usueful a reproducer project
for the slow down of performance that i observed

Thanks.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Performance degradation in case of high volumes

Posted by Antonio Conforti <an...@sia.eu>.
Hello Ilya,
you are right from 65681 to 82206 and the corresponding duration of
checkpoint from 1,6 sec to 9,6 sec it is not catastrophic. I just wanted to
highlight a noticeable increase in dirty pages in a short time despite the
constant rate, suggesting that in the long run the performance would be
degraded. Unfortunately I could not finish the test but right now I'm doing
a complete test: 4000 entries per second trying to get to 60 million
entries.
I'll post you results as soon as possible.

Thanks again for your collaboration!

Antonio



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Performance degradation in case of high volumes

Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello?

What's delay of checkpoint?

I fail to see how 82206 is catastrophically worse than 65681. To me it
sounds like sensible increase.

Regards,
-- 
Ilya Kasnacheev


чт, 28 февр. 2019 г. в 19:01, Antonio Conforti <an...@sia.eu>:

> Hello Ilya,
>
>
> if I well interpret the log for example of node with consitentid 7 what I
> see is a constant increment of dirty pages:
>
> 2019-02-27 17:04:42.206  INFO 41153 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=b3d5b34c-60d3-4c39-af0f-5de6d41686f3, pages=65681,
> markPos=FileWALPointer [idx=16, fileOff=24231459, len=75645],
> walSegmentsCleared=6, walSegmentsCovered=[10 - 15], markDuration=74ms,
> pagesWrite=1429ms, fsync=101ms, total=1604ms]
> 2019-02-27 17:07:42.244  INFO 41153 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=83cc0c0a-9427-465d-a290-f8a56c366528, pages=69139,
> markPos=FileWALPointer [idx=22, fileOff=41402244, len=75645],
> walSegmentsCleared=6, walSegmentsCovered=[16 - 21], markDuration=26ms,
> pagesWrite=1521ms, fsync=101ms, total=1648ms]
> 2019-02-27 17:10:42.553  INFO 41153 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=4fe0396f-5ece-4016-a269-119d5ecd8d16, pages=72896,
> markPos=FileWALPointer [idx=29, fileOff=6850845, len=75645],
> walSegmentsCleared=12, walSegmentsCovered=[22 - 28], markDuration=28ms,
> pagesWrite=1713ms, fsync=207ms, total=1948ms]
> 2019-02-27 17:13:45.214  INFO 41153 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=a309e6a2-865d-439e-ab6d-11ef9641c8b0, pages=75381,
> markPos=FileWALPointer [idx=35, fileOff=48279077, len=75645],
> walSegmentsCleared=7, walSegmentsCovered=[29 - 34], markDuration=44ms,
> pagesWrite=4446ms, fsync=112ms, total=4602ms]
> 2019-02-27 17:16:42.728  INFO 41153 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=0299f7d3-8ed0-4f0b-8499-14e46d3bef13, pages=78944,
> markPos=FileWALPointer [idx=42, fileOff=39209416, len=75645],
> walSegmentsCleared=6, walSegmentsCovered=[35 - 41], markDuration=28ms,
> pagesWrite=1989ms, fsync=95ms, total=2112ms]
> 2019-02-27 17:19:40.829  INFO 41153 --- [r-stripe-10-#27]
> o.a.i.i.p.c.p.pagemem.PageMemoryImpl     : Throttling is applied to page
> modifications [percentOfPartTime=0.31, markDirty=22777 pages/sec,
> checkpointWrite=11469 pages/sec, estIdealMarkDirty=226519 pages/sec,
> curDirty=0.00, maxDirty=0.07, avgParkTime=122057 ns, pages: (total=82206,
> evicted=0, written=10035, synced=0, cpBufUsed=27, cpBufTotal=518215)]
> 2019-02-27 17:19:50.524  INFO 41153 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=f323c2fd-d358-469e-8e12-d2ae1391a546, pages=82206,
> markPos=FileWALPointer [idx=49, fileOff=42053202, len=75645],
> walSegmentsCleared=8, walSegmentsCovered=[42 - 48], markDuration=27ms,
> pagesWrite=9749ms, fsync=123ms, total=9899ms]
> 2019-02-27 17:22:40.885  INFO 41153 --- [r-stripe-11-#28]
> o.a.i.i.p.c.p.pagemem.PageMemoryImpl     : Throttling is applied to page
> modifications [percentOfPartTime=0.23, markDirty=14449 pages/sec,
> checkpointWrite=6729 pages/sec, estIdealMarkDirty=166922 pages/sec,
> curDirty=0.00, maxDirty=0.09, avgParkTime=108997 ns, pages: (total=68092,
> evicted=0, written=13266, synced=0, cpBufUsed=12, cpBufTotal=518215)]
>
>
> I try to explain better:
>
> I see at every finished checkpoint a constant increease of dirty pages and
> a
> consequent duration increase of checkpoint:
>
> 1) at 17:04:42.206 dirty pages are 65681 with a delay of checkpoint of
> 1604ms
> 2) at 17:19:50.524 dirty pages are 82206 with a delay of checkpoint of
> 9899ms
>
>
> So in only 15 minutes with constant rate of 4000 entry/sec I observe a
> increment of delay of about 9 times respect to start of simulation and the
> consequent apllication of throttling how reported in the log at 17:19:40.
>
> I expected a constant number of dirty page and a consequent constant delay
> of checkpoint.
>
> Do you have any suggestion about that?
>
> Thanks a lot.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Performance degradation in case of high volumes

Posted by Antonio Conforti <an...@sia.eu>.
Hello Ilya,


if I well interpret the log for example of node with consitentid 7 what I
see is a constant increment of dirty pages:

2019-02-27 17:04:42.206  INFO 41153 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=b3d5b34c-60d3-4c39-af0f-5de6d41686f3, pages=65681,
markPos=FileWALPointer [idx=16, fileOff=24231459, len=75645],
walSegmentsCleared=6, walSegmentsCovered=[10 - 15], markDuration=74ms,
pagesWrite=1429ms, fsync=101ms, total=1604ms]
2019-02-27 17:07:42.244  INFO 41153 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=83cc0c0a-9427-465d-a290-f8a56c366528, pages=69139,
markPos=FileWALPointer [idx=22, fileOff=41402244, len=75645],
walSegmentsCleared=6, walSegmentsCovered=[16 - 21], markDuration=26ms,
pagesWrite=1521ms, fsync=101ms, total=1648ms]
2019-02-27 17:10:42.553  INFO 41153 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=4fe0396f-5ece-4016-a269-119d5ecd8d16, pages=72896,
markPos=FileWALPointer [idx=29, fileOff=6850845, len=75645],
walSegmentsCleared=12, walSegmentsCovered=[22 - 28], markDuration=28ms,
pagesWrite=1713ms, fsync=207ms, total=1948ms]
2019-02-27 17:13:45.214  INFO 41153 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=a309e6a2-865d-439e-ab6d-11ef9641c8b0, pages=75381,
markPos=FileWALPointer [idx=35, fileOff=48279077, len=75645],
walSegmentsCleared=7, walSegmentsCovered=[29 - 34], markDuration=44ms,
pagesWrite=4446ms, fsync=112ms, total=4602ms]
2019-02-27 17:16:42.728  INFO 41153 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=0299f7d3-8ed0-4f0b-8499-14e46d3bef13, pages=78944,
markPos=FileWALPointer [idx=42, fileOff=39209416, len=75645],
walSegmentsCleared=6, walSegmentsCovered=[35 - 41], markDuration=28ms,
pagesWrite=1989ms, fsync=95ms, total=2112ms]
2019-02-27 17:19:40.829  INFO 41153 --- [r-stripe-10-#27]
o.a.i.i.p.c.p.pagemem.PageMemoryImpl     : Throttling is applied to page
modifications [percentOfPartTime=0.31, markDirty=22777 pages/sec,
checkpointWrite=11469 pages/sec, estIdealMarkDirty=226519 pages/sec,
curDirty=0.00, maxDirty=0.07, avgParkTime=122057 ns, pages: (total=82206,
evicted=0, written=10035, synced=0, cpBufUsed=27, cpBufTotal=518215)]
2019-02-27 17:19:50.524  INFO 41153 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=f323c2fd-d358-469e-8e12-d2ae1391a546, pages=82206,
markPos=FileWALPointer [idx=49, fileOff=42053202, len=75645],
walSegmentsCleared=8, walSegmentsCovered=[42 - 48], markDuration=27ms,
pagesWrite=9749ms, fsync=123ms, total=9899ms]
2019-02-27 17:22:40.885  INFO 41153 --- [r-stripe-11-#28]
o.a.i.i.p.c.p.pagemem.PageMemoryImpl     : Throttling is applied to page
modifications [percentOfPartTime=0.23, markDirty=14449 pages/sec,
checkpointWrite=6729 pages/sec, estIdealMarkDirty=166922 pages/sec,
curDirty=0.00, maxDirty=0.09, avgParkTime=108997 ns, pages: (total=68092,
evicted=0, written=13266, synced=0, cpBufUsed=12, cpBufTotal=518215)]


I try to explain better:

I see at every finished checkpoint a constant increease of dirty pages and a
consequent duration increase of checkpoint: 

1) at 17:04:42.206 dirty pages are 65681 with a delay of checkpoint of 
1604ms
2) at 17:19:50.524 dirty pages are 82206 with a delay of checkpoint of 
9899ms


So in only 15 minutes with constant rate of 4000 entry/sec I observe a
increment of delay of about 9 times respect to start of simulation and the
consequent apllication of throttling how reported in the log at 17:19:40.

I expected a constant number of dirty page and a consequent constant delay
of checkpoint.

Do you have any suggestion about that?

Thanks a lot. 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Performance degradation in case of high volumes

Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello Antonio!

I have reviewed your logs but don't understand what is the issue here. Can
you please clarify what is the problem once the warning went away?

Of course you should expect eventual slowdown since the height of B-tree
will still increase with time.

Regards,
-- 
Ilya Kasnacheev


чт, 28 февр. 2019 г. в 15:27, Antonio Conforti <an...@sia.eu>:

> Hello Ilya,
> there was a misunderstanding.
> The warn log message was the explanation how I set the value 66 as max
> payload.
> After setting the environment variable IGNITE_MAX_INDEX_PAYLOAD_SIZE the
> warning message disappeared.
> But the test seems failed.
> You can see my evaluations in the message above.
>
> Thanks a lot and I look forward to hearing from you.
>
> Regards
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Performance degradation in case of high volumes

Posted by Antonio Conforti <an...@sia.eu>.
Hello Ilya,
there was a misunderstanding.
The warn log message was the explanation how I set the value 66 as max
payload.
After setting the environment variable IGNITE_MAX_INDEX_PAYLOAD_SIZE the
warning message disappeared.
But the test seems failed.
You can see my evaluations in the message above.

Thanks a lot and I look forward to hearing from you.

Regards




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Performance degradation in case of high volumes

Posted by "ilya.kasnacheev" <il...@gmail.com>.
Hello!

I'm afraid you will have to increase IGNITE_MAX_INDEX_PAYLOAD_SIZE until you
no longer see "Indexed columns of a row cannot be fully inlined" warning.
Can you try?

Alternatively you can try specifying it with
cacheCfg.setSqlIndexMaxInlineSize().

Regards,



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Performance degradation in case of high volumes

Posted by Antonio Conforti <an...@sia.eu>.
Hello Ilya.
Thank you so much for your answer.
I tried your solution in the same scenario:
-	PARTITIONED cache, SQL indexed by key and by a field (type long, the last
key field) descending
-	checkpoint frequency restored to default
-	4000 msg/sec rate (fixed)
I set the environment variable IGNITE_MAX_INDEX_PAYLOAD_SIZE to 66:
(WARN 59726 --- [0-#17%Datafeed%] o.a.i.i.p.query.h2.database.H2TreeIndex  :
<DF_CMF_QUOTE> Indexed columns of a row cannot be fully inlined into index
what may lead to slowdown due to additional data page reads, increase index
inline size if needed (set system property IGNITE_MAX_INDEX_PAYLOAD_SIZE
with recommended size (be aware it will be used by default for all indexes
without explicit inline size)) [cacheName=IGN_DF_CMF_QUOTE,
tableName=DF_CMF_QUOTE, idxName=_key_PK, idxCols=(_KEY), idxType=PRIMARY
KEY, curSize=10, recommendedInlineSize=66])
in all cluster nodes and I ran the test again.
We had to stop the test after 3 milion messages and what I could notice was
that the checkpoints collected a number of pages that grew over and over.
This increment grew much faster than in previous tests
(IGNITE_MAX_INDEX_PAYLOAD_SIZE not set). 
So, I couldn't detect the performance degradation (because I had to stop the
test necessarily) but the increasing number of pages in every checkpoints
let me expect it will happen.

I attach the server logs of this test.

log_ignite_190227_setIndexPayload.gz
<http://apache-ignite-users.70518.x6.nabble.com/file/t2315/log_ignite_190227_setIndexPayload.gz>  


What do you think about ?

Thanks a lot

Regards




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Performance degradation in case of high volumes

Posted by Justin Ji <bi...@foxmail.com>.
Thank for your reply, the project was created, you can refer to :
https://github.com/RedBlackTreei/streamer.git

Related post:
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Data-Streamer-Hung-after-a-period-td21161.html

Looking forward to your reply!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Performance degradation in case of high volumes

Posted by Justin Ji <bi...@foxmail.com>.
Thank for your reply, the project was created, you can refer to :
https://github.com/RedBlackTreei/streamer.git

Related post:
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Data-Streamer-Hung-after-a-period-td21161.html

Looking forward to your reply!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Performance degradation in case of high volumes

Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!

It's hard to say. Maybe the problem is caused by some other index.

Can you make a reproducer project which will exhibit this behavior so that
we could check?

Regards,
-- 
Ilya Kasnacheev


чт, 28 февр. 2019 г. в 17:41, Justin Ji <bi...@foxmail.com>:

> Ilya -
>
> I have tried to increase QueryIndex.setInlineSize and
> CacheConfiguration.setSqlIndexMaxInlineSize to 128 256 and 512, but the
> performace became worse.
>
> Do I miss some configuraiton?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Performance degradation in case of high volumes

Posted by Justin Ji <bi...@foxmail.com>.
Ilya -

I have tried to increase QueryIndex.setInlineSize and
CacheConfiguration.setSqlIndexMaxInlineSize to 128 256 and 512, but the
performace became worse.

Do I miss some configuraiton?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Performance degradation in case of high volumes

Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!

Since your key is composite, it's hard to estimate its length (BinaryObject
has a lot of overhead). It's recommended to increase
IGNITE_MAX_INDEX_PAYLOAD_SIZE
until you no longer see this problem. Please try e.g. 128.

You can also try specifying that via setSqlIndexMaxInlineSize().

Regards,
-- 
Ilya Kasnacheev


чт, 28 февр. 2019 г. в 16:21, BinaryTree <bi...@foxmail.com>:

> Hi Ilya -
> First of all, thank for your reply!
> Here is my cache configuration:
>
> private static CacheConfiguration<DpKey, BinaryObject> getCacheConfiguration(IgniteConfiguration cfg) {
>
>     CacheConfiguration<DpKey, BinaryObject> cacheCfg = new CacheConfiguration();
>     cacheCfg.setName(IgniteCacheKey.DATA_POINT_NEW.getCode());
>     cacheCfg.setCacheMode(CacheMode.PARTITIONED);
>     cacheCfg.setBackups(1);
>     cacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
>     cacheCfg.setDataRegionName(Constants.FIVE_GB_PERSISTENCE_REGION);
>     cacheCfg.setCacheStoreFactory(FactoryBuilder.factoryOf(DataPointCacheStore.class));
>     cacheCfg.setWriteThrough(true);
>     cacheCfg.setWriteBehindEnabled(true);
>     cacheCfg.setWriteBehindFlushThreadCount(2);
>     cacheCfg.setWriteBehindFlushFrequency(15 * 1000);
>     cacheCfg.setWriteBehindFlushSize(409600);
>     cacheCfg.setWriteBehindBatchSize(1024);
>     cacheCfg.setStoreKeepBinary(true);
>     cacheCfg.setQueryParallelism(16);
>
>     //2M
>     cacheCfg.setRebalanceBatchSize(2 * 1024 * 1024);
>     cacheCfg.setRebalanceThrottle(100);
>
>     cacheCfg.setSqlIndexMaxInlineSize(256);
>
>     List<QueryEntity> entities = getQueryEntities();
>     cacheCfg.setQueryEntities(entities);
>
>     CacheKeyConfiguration cacheKeyConfiguration = new CacheKeyConfiguration(DpKey.class);
>     cacheCfg.setKeyConfiguration(cacheKeyConfiguration);
>
>     RendezvousAffinityFunction affinityFunction = new RendezvousAffinityFunction();
>     affinityFunction.setPartitions(128);
>     affinityFunction.setExcludeNeighbors(true);
>     cacheCfg.setAffinity(affinityFunction);
>
>
> cfg.setCacheConfiguration(cacheCfg);
> return cacheCfg;
> }
>
>
> private static List<QueryEntity> getQueryEntities() {
>     List<QueryEntity> entities = Lists.newArrayList();
>
>     //配置可见(可被查询)字段
>     QueryEntity entity = new QueryEntity(DpKey.class.getName(), DpCache.class.getName());
>     entity.setTableName(IgniteTableKey.T_DATA_POINT_NEW.getCode());
>
>     LinkedHashMap<String, String> map = new LinkedHashMap<>();
>     map.put("id", "java.lang.String");
>     map.put("gmtCreate", "java.lang.Long");
>     map.put("gmtModified", "java.lang.Long");
>     map.put("devId", "java.lang.String");
>     map.put("dpId", "java.lang.Integer");
>     map.put("code", "java.lang.String");
>     map.put("name", "java.lang.String");
>     map.put("customName", "java.lang.String");
>     map.put("mode", "java.lang.String");
>     map.put("type", "java.lang.String");
>     map.put("value", "java.lang.String");
>     map.put("rawValue", byte[].class.getName());
>     map.put("time", "java.lang.Long");
>     map.put("status", "java.lang.Boolean");
>     map.put("uuid", "java.lang.String");
>
>     entity.setFields(map);
>
>     //配置索引信息
>     QueryIndex devIdIdx = new QueryIndex("devId");
>     devIdIdx.setName("idx_devId");
>     devIdIdx.setInlineSize(32);
>     List<QueryIndex> indexes = Lists.newArrayList(devIdIdx);
>     entity.setIndexes(indexes);
>
>     entities.add(entity);
>
>     return entities;
> }
>
> public class DpKey implements Serializable {
>     private String key;
>     @AffinityKeyMapped
>     private String devId;
>
>     public DpKey() {
>     }
>
>     public DpKey(String key, String devId) {
>         this.key = key;
>         this.devId = devId;
>     }
>
>     public String getKey() {
>         return this.key;
>     }
>
>     public void setKey(String key) {
>         this.key = key;
>     }
>
>     public String getDevId() {
>         return this.devId;
>     }
>
>     public void setDevId(String devId) {
>         this.devId = devId;
>     }
>
>     public boolean equals(Object o) {
>         if (this == o) {
>             return true;
>         } else if (o != null && this.getClass() == o.getClass()) {
>             DpKey key = (DpKey)o;
>             return this.key.equals(key.key);
>         } else {
>             return false;
>         }
>     }
>
>     public int hashCode() {
>         return this.key.hashCode();
>     }
> }
>
> And I have described my issue in this post and some tests I have done :
>
> http://apache-ignite-users.70518.x6.nabble.com/Ignite-Data-Streamer-Hung-after-a-period-td21161.html
>
> ------------------ 原始邮件 ------------------
> *发件人:* "ilya.kasnacheev"<il...@gmail.com>;
> *发送时间:* 2019年2月28日(星期四) 晚上9:03
> *收件人:* "user"<us...@ignite.apache.org>;
> *主题:* Re: Performance degradation in case of high volumes
>
> Hello Justin!
>
> Ignite 2.6 does have IGNITE_MAX_INDEX_PAYLOAD_SIZE system property.
>
> We are talking about primary key here. What is your primary key type? What
> other indexes do you have? Can you provide complete configuration for
> affected tables (including POJOs if applicable?)
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 28 февр. 2019 г. в 15:29, Justin Ji <bi...@foxmail.com>:
>
>> Ilya -
>>
>> I use ignite 2.6.0, does not have IGNITE_MAX_INDEX_PAYLOAD_SIZE system
>> property.
>> But our index field has a fixed length:25 characters, so where can I find
>> the algorithm to calculate the 'index inline size'.
>>
>> Looking forward to your reply.
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

回复: Performance degradation in case of high volumes

Posted by BinaryTree <bi...@foxmail.com>.
Hi Ilya - 
First of all, thank for your reply!
Here is my cache configuration:
private static CacheConfiguration<DpKey, BinaryObject> getCacheConfiguration(IgniteConfiguration cfg) {

    CacheConfiguration<DpKey, BinaryObject> cacheCfg = new CacheConfiguration();
    cacheCfg.setName(IgniteCacheKey.DATA_POINT_NEW.getCode());
    cacheCfg.setCacheMode(CacheMode.PARTITIONED);
    cacheCfg.setBackups(1);
    cacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
    cacheCfg.setDataRegionName(Constants.FIVE_GB_PERSISTENCE_REGION);
    cacheCfg.setCacheStoreFactory(FactoryBuilder.factoryOf(DataPointCacheStore.class));
    cacheCfg.setWriteThrough(true);
    cacheCfg.setWriteBehindEnabled(true);
    cacheCfg.setWriteBehindFlushThreadCount(2);
    cacheCfg.setWriteBehindFlushFrequency(15 * 1000);
    cacheCfg.setWriteBehindFlushSize(409600);
    cacheCfg.setWriteBehindBatchSize(1024);
    cacheCfg.setStoreKeepBinary(true);
    cacheCfg.setQueryParallelism(16);

    //2M
    cacheCfg.setRebalanceBatchSize(2 * 1024 * 1024);
    cacheCfg.setRebalanceThrottle(100);

    cacheCfg.setSqlIndexMaxInlineSize(256);

    List<QueryEntity> entities = getQueryEntities();
    cacheCfg.setQueryEntities(entities);

    CacheKeyConfiguration cacheKeyConfiguration = new CacheKeyConfiguration(DpKey.class);
    cacheCfg.setKeyConfiguration(cacheKeyConfiguration);

    RendezvousAffinityFunction affinityFunction = new RendezvousAffinityFunction();
    affinityFunction.setPartitions(128);
    affinityFunction.setExcludeNeighbors(true);
    cacheCfg.setAffinity(affinityFunction);
    cfg.setCacheConfiguration(cacheCfg);
    return cacheCfg;
}


private static List<QueryEntity> getQueryEntities() {
    List<QueryEntity> entities = Lists.newArrayList();

    //配置可见(可被查询)字段
    QueryEntity entity = new QueryEntity(DpKey.class.getName(), DpCache.class.getName());
    entity.setTableName(IgniteTableKey.T_DATA_POINT_NEW.getCode());

    LinkedHashMap<String, String> map = new LinkedHashMap<>();
    map.put("id", "java.lang.String");
    map.put("gmtCreate", "java.lang.Long");
    map.put("gmtModified", "java.lang.Long");
    map.put("devId", "java.lang.String");
    map.put("dpId", "java.lang.Integer");
    map.put("code", "java.lang.String");
    map.put("name", "java.lang.String");
    map.put("customName", "java.lang.String");
    map.put("mode", "java.lang.String");
    map.put("type", "java.lang.String");
    map.put("value", "java.lang.String");
    map.put("rawValue", byte[].class.getName());
    map.put("time", "java.lang.Long");
    map.put("status", "java.lang.Boolean");
    map.put("uuid", "java.lang.String");

    entity.setFields(map);

    //配置索引信息
    QueryIndex devIdIdx = new QueryIndex("devId");
    devIdIdx.setName("idx_devId");
    devIdIdx.setInlineSize(32);
    List<QueryIndex> indexes = Lists.newArrayList(devIdIdx);
    entity.setIndexes(indexes);

    entities.add(entity);

    return entities;
}
public class DpKey implements Serializable {
    private String key;
    @AffinityKeyMapped
    private String devId;

    public DpKey() {
    }

    public DpKey(String key, String devId) {
        this.key = key;
        this.devId = devId;
    }

    public String getKey() {
        return this.key;
    }

    public void setKey(String key) {
        this.key = key;
    }

    public String getDevId() {
        return this.devId;
    }

    public void setDevId(String devId) {
        this.devId = devId;
    }

    public boolean equals(Object o) {
        if (this == o) {
            return true;
        } else if (o != null && this.getClass() == o.getClass()) {
            DpKey key = (DpKey)o;
            return this.key.equals(key.key);
        } else {
            return false;
        }
    }

    public int hashCode() {
        return this.key.hashCode();
    }
}

And I have described my issue in this post and some tests I have done :
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Data-Streamer-Hung-after-a-period-td21161.html


------------------ 原始邮件 ------------------
发件人: "ilya.kasnacheev"<il...@gmail.com>;
发送时间: 2019年2月28日(星期四) 晚上9:03
收件人: "user"<us...@ignite.apache.org>;

主题: Re: Performance degradation in case of high volumes



Hello Justin!


Ignite 2.6 does have IGNITE_MAX_INDEX_PAYLOAD_SIZE system property.


We are talking about primary key here. What is your primary key type? What other indexes do you have? Can you provide complete configuration for affected tables (including POJOs if applicable?)


Regards,

-- 

Ilya Kasnacheev









чт, 28 февр. 2019 г. в 15:29, Justin Ji <bi...@foxmail.com>:

Ilya - 
 
 I use ignite 2.6.0, does not have IGNITE_MAX_INDEX_PAYLOAD_SIZE system
 property.
 But our index field has a fixed length:25 characters, so where can I find
 the algorithm to calculate the 'index inline size'.
 
 Looking forward to your reply.
 
 
 
 --
 Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Performance degradation in case of high volumes

Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello Justin!

Ignite 2.6 does have IGNITE_MAX_INDEX_PAYLOAD_SIZE system property.

We are talking about primary key here. What is your primary key type? What
other indexes do you have? Can you provide complete configuration for
affected tables (including POJOs if applicable?)

Regards,
-- 
Ilya Kasnacheev


чт, 28 февр. 2019 г. в 15:29, Justin Ji <bi...@foxmail.com>:

> Ilya -
>
> I use ignite 2.6.0, does not have IGNITE_MAX_INDEX_PAYLOAD_SIZE system
> property.
> But our index field has a fixed length:25 characters, so where can I find
> the algorithm to calculate the 'index inline size'.
>
> Looking forward to your reply.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Performance degradation in case of high volumes

Posted by Justin Ji <bi...@foxmail.com>.
Ilya - 

I use ignite 2.6.0, does not have IGNITE_MAX_INDEX_PAYLOAD_SIZE system
property.
But our index field has a fixed length:25 characters, so where can I find
the algorithm to calculate the 'index inline size'.

Looking forward to your reply.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Performance degradation in case of high volumes

Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!

It is possible that your primary key would not fit into index and walking
index tree will put unnecessary strain on persistent page memory.

On 2.7 there will be a warning about that in log.

Please try to tune IGNITE_MAX_INDEX_PAYLOAD_SIZE system property since it's
the only available means of changing that value. Try to put something line
32 or 64 there, see if it helps.

Regards,
-- 
Ilya Kasnacheev


вт, 26 февр. 2019 г. в 20:24, Antonio Conforti <an...@sia.eu>:

> Hello,
>
>
> I recap the scenario of benchmark:
>
> 1) Constant submission of 4000 entries per second where every entry is an
> add (the key contains a field updatetime and changes for every entry).
> 2) The benchmark starts with no data in cache and the entries are submitted
> from an Ignite client node in the cluster using the StreamVisitor.addData()
> method (located on HOST1).
> 3) The cluster is composed of a total of 8 ignite server nodes: server
> nodes
> with consistence ID 1,3,5,7 on HOST1 and server nodes with consistence ID
> 2,4,6,8 on HOST2;
>
>
> Below the general configuration:
>
> Cache configuration:
>
> 1.            TRANSACTIONAL
> 2.            partitioned
> 3.            with backup 1 (and affinity with exclude neighbours enabled)
> 4.            write synchronization mode FULL_ASYNC
> 5.            indexed on key and value (and enabled to SQL inquiry)
>
>
> We have also configured:
> 1.            failureDetectionTimeout to 120000msec
> 2.            Data region (only 1):
> a.            Persistence enabled
> b.            max size 8 GB
> c.            checkpointPageBufferSize 2 GB
> 3.            WAL mode LOG_ONLY
> 4.            disabled WAL archiving (WAL path and the WAL archive path to
> the same
> value)
> 5.            Pages Writes Throttling enabled
>
>
> I run also a second scenario with frequency set to 600000 (10minutes) as
> suggested and with direct IO enabled and WAL mode set to NONE.
>
>
> For problem of space you can find logs and configuration file of only
> server
> node 1 attached for both scenarios as described below:
>
> 1) folder 20190222: log and config for scenario 1)
> 2) folder 20190225: log and config for scenario 2)
> 3) IGN_DF_CMF_QUOTE_PK.java: key entity inserted in cache
> 4) IGN_DF_CMF_QUOTE.java: entity data inserted in cache
>
> case-Ignite.zip
> <http://apache-ignite-users.70518.x6.nabble.com/file/t2315/case-Ignite.zip>
>
>
> If you need another benchmark with specific configuration, just let me
> know.
>
> Thanks.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Performance degradation in case of high volumes

Posted by Antonio Conforti <an...@sia.eu>.
Hello,


I recap the scenario of benchmark:

1) Constant submission of 4000 entries per second where every entry is an
add (the key contains a field updatetime and changes for every entry).
2) The benchmark starts with no data in cache and the entries are submitted
from an Ignite client node in the cluster using the StreamVisitor.addData()
method (located on HOST1).
3) The cluster is composed of a total of 8 ignite server nodes: server nodes
with consistence ID 1,3,5,7 on HOST1 and server nodes with consistence ID
2,4,6,8 on HOST2;


Below the general configuration:

Cache configuration:

1.            TRANSACTIONAL 
2.            partitioned 
3.            with backup 1 (and affinity with exclude neighbours enabled) 
4.            write synchronization mode FULL_ASYNC 
5.            indexed on key and value (and enabled to SQL inquiry) 


We have also configured: 
1.            failureDetectionTimeout to 120000msec 
2.            Data region (only 1): 
a.            Persistence enabled 
b.            max size 8 GB 
c.            checkpointPageBufferSize 2 GB 
3.            WAL mode LOG_ONLY 
4.            disabled WAL archiving (WAL path and the WAL archive path to
the same 
value) 
5.            Pages Writes Throttling enabled 


I run also a second scenario with frequency set to 600000 (10minutes) as
suggested and with direct IO enabled and WAL mode set to NONE.


For problem of space you can find logs and configuration file of only server
node 1 attached for both scenarios as described below:

1) folder 20190222: log and config for scenario 1)
2) folder 20190225: log and config for scenario 2)
3) IGN_DF_CMF_QUOTE_PK.java: key entity inserted in cache
4) IGN_DF_CMF_QUOTE.java: entity data inserted in cache

case-Ignite.zip
<http://apache-ignite-users.70518.x6.nabble.com/file/t2315/case-Ignite.zip>  

If you need another benchmark with specific configuration, just let me know.

Thanks.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Performance degradation in case of high volumes

Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!

Can you please provide full log? It's hard to say what is going on here.
Can you please also share your Ignite/cache configuration and describe your
work load?

Regards,
-- 
Ilya Kasnacheev


пн, 25 февр. 2019 г. в 17:32, Dodong Juan <do...@gmail.com>:

> I did observe the same thing with ignite and was not able to resolve it at
> all.
> So I am very interested on a resolution for this too.
>
> > On Feb 25, 2019, at 9:30 AM, Antonio Conforti <an...@sia.eu>
> wrote:
> >
> > Hello,
> >
> > first of all thanks for your advice.
> >
> > Before reading your post I runned another session (at 15:45 about) of
> test
> > with the same rate before changing the frequency of checkpoint you
> suggested
> > but with data already loaded.
> >
> >
> > I observed that performance degrades suddenly and not after a while
> (about
> > 45 minutes) as observed in my first session when the cache was empty.
> > Below you can see  the statistics:
> >
> > 2019-02-22 15:49:03.992  INFO 5271 --- [oint-thread-#67]
> > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> > [cpId=ab04c732-a72c-4398-8a09-31f05243a7ad, pages=129721,
> > markPos=FileWALPointer [idx=470, fileOff=35838781, len=79426],
> > walSegmentsCleared=11, walSegmentsCovered=[460 - 469],
> markDuration=387ms,
> > pagesWrite=1149ms, fsync=227229ms, total=228765ms]
> > 2019-02-22 15:53:25.681  INFO 5271 --- [oint-thread-#67]
> > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> > [cpId=cf6e6d68-ac4d-4445-becb-b7bcbb5aee32, pages=178943,
> > markPos=FileWALPointer [idx=484, fileOff=61358243, len=79426],
> > walSegmentsCleared=14, walSegmentsCovered=[470 - 483],
> markDuration=1316ms,
> > pagesWrite=1043ms, fsync=259329ms, total=261688ms]
> > 2019-02-22 15:56:07.878  INFO 5271 --- [oint-thread-#67]
> > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> > [cpId=86154f6f-907e-4972-b0f1-f86be06c3a40, pages=44514,
> > markPos=FileWALPointer [idx=487, fileOff=66267301, len=79426],
> > walSegmentsCleared=0, walSegmentsCovered=[484 - 486], markDuration=888ms,
> > pagesWrite=239ms, fsync=161070ms, total=162197ms]
> >
> > What looks strange in this second session is that I see a delay between
> 228
> > and 260 sec to write 129721 / 178943 pages.
> > I don't observe this behaviour when I start from scratch (with no data in
> > cache): it usually takes about 45 minutes before observing this
> performance
> > degradation.
> > Any clues on what can be the cause?
> >
> >
> > I started today another test session with the suggested frequency
> > (checkpointingFrequency=600000) and I also configured direct IO and set
> the
> > WAL mode to NONE as suggested in tuning performance but the result
> observed
> > is quite the same
> >
> >
> >
> > 2019-02-25 09:34:02.284  INFO 1208 --- [oint-thread-#66]
> > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> > [cpId=a9423ba9-9d93-40a3-bedb-7379a617f716, pages=78,
> markPos=FileWALPointer
> > [idx=0, fileOff=0, len=0], walSegmentsCleared=0, walSegmentsCovered=[],
> > markDuration=13ms, pagesWrite=15ms, fsync=11ms, total=39ms]
> > 2019-02-25 10:04:04.082  INFO 1208 --- [oint-thread-#66]
> > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> > [cpId=bd76a704-e125-42e1-b2a8-d285cf15246d, pages=52252,
> > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> > walSegmentsCovered=[], markDuration=144ms, pagesWrite=1560ms,
> fsync=123ms,
> > total=1827ms]
> > 2019-02-25 10:14:27.275  INFO 1208 --- [oint-thread-#66]
> > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> > [cpId=a10f5483-050c-4678-98bd-deca7695e782, pages=368810,
> > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> > walSegmentsCovered=[], markDuration=323ms, pagesWrite=24669ms,
> fsync=21ms,
> > total=25013ms]
> > 2019-02-25 10:24:55.492  INFO 1208 --- [oint-thread-#66]
> > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> > [cpId=bfab59c6-a1a0-4391-92e5-a23b6c416625, pages=382555,
> > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> > walSegmentsCovered=[], markDuration=225ms, pagesWrite=52912ms,
> fsync=93ms,
> > total=53230ms]
> > 2019-02-25 10:35:22.512  INFO 1208 --- [oint-thread-#66]
> > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> > [cpId=feff29e3-db80-4002-82f8-2011e4eca12e, pages=436129,
> > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> > walSegmentsCovered=[], markDuration=70ms, pagesWrite=80164ms, fsync=7ms,
> > total=80241ms]
> > 2019-02-25 10:45:34.524  INFO 1208 --- [oint-thread-#66]
> > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> > [cpId=7e3f181b-21d8-4dd1-a018-09711142caf4, pages=384415,
> > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> > walSegmentsCovered=[], markDuration=56ms, pagesWrite=92186ms, fsync=3ms,
> > total=92245ms]
> > 2019-02-25 10:55:43.139  INFO 1208 --- [oint-thread-#66]
> > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> > [cpId=a77d48d8-f806-4a48-bfbf-d1afac7d16e3, pages=161643,
> > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> > walSegmentsCovered=[], markDuration=73ms, pagesWrite=100765ms,
> fsync=16ms,
> > total=100854ms]
> > 2019-02-25 11:06:04.437  INFO 1208 --- [oint-thread-#66]
> > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> > [cpId=dbf6a8ab-1c47-4cdd-96f4-fec852dc4fe0, pages=129477,
> > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> > walSegmentsCovered=[], markDuration=60ms, pagesWrite=121933ms,
> fsync=154ms,
> > total=122147ms]
> > 2019-02-25 11:16:08.333  INFO 1208 --- [oint-thread-#66]
> > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> > [cpId=74924992-3dc4-42db-971c-b9030f41d0c5, pages=97511,
> > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> > walSegmentsCovered=[], markDuration=1676ms, pagesWrite=124336ms,
> fsync=25ms,
> > total=126037ms]
> > 2019-02-25 11:18:53.622  INFO 1208 --- [oint-thread-#66]
> > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> > [cpId=7c60efc2-4007-4be2-b3c8-3912d2cbd651, pages=19277,
> > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> > walSegmentsCovered=[], markDuration=12ms, pagesWrite=47883ms,
> fsync=344ms,
> > total=48239ms]
> >
> >
> > Do you have any other suggestions considering also the behaviour
> described
> > above?
> >
> > Thanks,
> > Antonio
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>

Re: Performance degradation in case of high volumes

Posted by Dodong Juan <do...@gmail.com>.
I did observe the same thing with ignite and was not able to resolve it at all.
So I am very interested on a resolution for this too.

> On Feb 25, 2019, at 9:30 AM, Antonio Conforti <an...@sia.eu> wrote:
> 
> Hello,
> 
> first of all thanks for your advice.
> 
> Before reading your post I runned another session (at 15:45 about) of test
> with the same rate before changing the frequency of checkpoint you suggested
> but with data already loaded.
> 
> 
> I observed that performance degrades suddenly and not after a while (about
> 45 minutes) as observed in my first session when the cache was empty.
> Below you can see  the statistics:
> 
> 2019-02-22 15:49:03.992  INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=ab04c732-a72c-4398-8a09-31f05243a7ad, pages=129721,
> markPos=FileWALPointer [idx=470, fileOff=35838781, len=79426],
> walSegmentsCleared=11, walSegmentsCovered=[460 - 469], markDuration=387ms,
> pagesWrite=1149ms, fsync=227229ms, total=228765ms]
> 2019-02-22 15:53:25.681  INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=cf6e6d68-ac4d-4445-becb-b7bcbb5aee32, pages=178943,
> markPos=FileWALPointer [idx=484, fileOff=61358243, len=79426],
> walSegmentsCleared=14, walSegmentsCovered=[470 - 483], markDuration=1316ms,
> pagesWrite=1043ms, fsync=259329ms, total=261688ms]
> 2019-02-22 15:56:07.878  INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=86154f6f-907e-4972-b0f1-f86be06c3a40, pages=44514,
> markPos=FileWALPointer [idx=487, fileOff=66267301, len=79426],
> walSegmentsCleared=0, walSegmentsCovered=[484 - 486], markDuration=888ms,
> pagesWrite=239ms, fsync=161070ms, total=162197ms]
> 
> What looks strange in this second session is that I see a delay between 228
> and 260 sec to write 129721 / 178943 pages.
> I don't observe this behaviour when I start from scratch (with no data in
> cache): it usually takes about 45 minutes before observing this performance
> degradation.
> Any clues on what can be the cause?
> 
> 
> I started today another test session with the suggested frequency
> (checkpointingFrequency=600000) and I also configured direct IO and set the
> WAL mode to NONE as suggested in tuning performance but the result observed
> is quite the same
> 
> 
> 
> 2019-02-25 09:34:02.284  INFO 1208 --- [oint-thread-#66]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=a9423ba9-9d93-40a3-bedb-7379a617f716, pages=78, markPos=FileWALPointer
> [idx=0, fileOff=0, len=0], walSegmentsCleared=0, walSegmentsCovered=[],
> markDuration=13ms, pagesWrite=15ms, fsync=11ms, total=39ms]
> 2019-02-25 10:04:04.082  INFO 1208 --- [oint-thread-#66]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=bd76a704-e125-42e1-b2a8-d285cf15246d, pages=52252,
> markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> walSegmentsCovered=[], markDuration=144ms, pagesWrite=1560ms, fsync=123ms,
> total=1827ms]
> 2019-02-25 10:14:27.275  INFO 1208 --- [oint-thread-#66]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=a10f5483-050c-4678-98bd-deca7695e782, pages=368810,
> markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> walSegmentsCovered=[], markDuration=323ms, pagesWrite=24669ms, fsync=21ms,
> total=25013ms]
> 2019-02-25 10:24:55.492  INFO 1208 --- [oint-thread-#66]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=bfab59c6-a1a0-4391-92e5-a23b6c416625, pages=382555,
> markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> walSegmentsCovered=[], markDuration=225ms, pagesWrite=52912ms, fsync=93ms,
> total=53230ms]
> 2019-02-25 10:35:22.512  INFO 1208 --- [oint-thread-#66]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=feff29e3-db80-4002-82f8-2011e4eca12e, pages=436129,
> markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> walSegmentsCovered=[], markDuration=70ms, pagesWrite=80164ms, fsync=7ms,
> total=80241ms]
> 2019-02-25 10:45:34.524  INFO 1208 --- [oint-thread-#66]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=7e3f181b-21d8-4dd1-a018-09711142caf4, pages=384415,
> markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> walSegmentsCovered=[], markDuration=56ms, pagesWrite=92186ms, fsync=3ms,
> total=92245ms]
> 2019-02-25 10:55:43.139  INFO 1208 --- [oint-thread-#66]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=a77d48d8-f806-4a48-bfbf-d1afac7d16e3, pages=161643,
> markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> walSegmentsCovered=[], markDuration=73ms, pagesWrite=100765ms, fsync=16ms,
> total=100854ms]
> 2019-02-25 11:06:04.437  INFO 1208 --- [oint-thread-#66]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=dbf6a8ab-1c47-4cdd-96f4-fec852dc4fe0, pages=129477,
> markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> walSegmentsCovered=[], markDuration=60ms, pagesWrite=121933ms, fsync=154ms,
> total=122147ms]
> 2019-02-25 11:16:08.333  INFO 1208 --- [oint-thread-#66]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=74924992-3dc4-42db-971c-b9030f41d0c5, pages=97511,
> markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> walSegmentsCovered=[], markDuration=1676ms, pagesWrite=124336ms, fsync=25ms,
> total=126037ms]
> 2019-02-25 11:18:53.622  INFO 1208 --- [oint-thread-#66]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=7c60efc2-4007-4be2-b3c8-3912d2cbd651, pages=19277,
> markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> walSegmentsCovered=[], markDuration=12ms, pagesWrite=47883ms, fsync=344ms,
> total=48239ms]
> 
> 
> Do you have any other suggestions considering also the behaviour described
> above?
> 
> Thanks,
> Antonio
> 
> 
> 
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/


Re: Performance degradation in case of high volumes

Posted by Antonio Conforti <an...@sia.eu>.
Hello,

first of all thanks for your advice.

Before reading your post I runned another session (at 15:45 about) of test
with the same rate before changing the frequency of checkpoint you suggested
but with data already loaded.


I observed that performance degrades suddenly and not after a while (about
45 minutes) as observed in my first session when the cache was empty.
Below you can see  the statistics:

2019-02-22 15:49:03.992  INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=ab04c732-a72c-4398-8a09-31f05243a7ad, pages=129721,
markPos=FileWALPointer [idx=470, fileOff=35838781, len=79426],
walSegmentsCleared=11, walSegmentsCovered=[460 - 469], markDuration=387ms,
pagesWrite=1149ms, fsync=227229ms, total=228765ms]
2019-02-22 15:53:25.681  INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=cf6e6d68-ac4d-4445-becb-b7bcbb5aee32, pages=178943,
markPos=FileWALPointer [idx=484, fileOff=61358243, len=79426],
walSegmentsCleared=14, walSegmentsCovered=[470 - 483], markDuration=1316ms,
pagesWrite=1043ms, fsync=259329ms, total=261688ms]
2019-02-22 15:56:07.878  INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=86154f6f-907e-4972-b0f1-f86be06c3a40, pages=44514,
markPos=FileWALPointer [idx=487, fileOff=66267301, len=79426],
walSegmentsCleared=0, walSegmentsCovered=[484 - 486], markDuration=888ms,
pagesWrite=239ms, fsync=161070ms, total=162197ms]

What looks strange in this second session is that I see a delay between 228
and 260 sec to write 129721 / 178943 pages.
I don't observe this behaviour when I start from scratch (with no data in
cache): it usually takes about 45 minutes before observing this performance
degradation.
Any clues on what can be the cause?


I started today another test session with the suggested frequency
(checkpointingFrequency=600000) and I also configured direct IO and set the
WAL mode to NONE as suggested in tuning performance but the result observed
is quite the same



2019-02-25 09:34:02.284  INFO 1208 --- [oint-thread-#66]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=a9423ba9-9d93-40a3-bedb-7379a617f716, pages=78, markPos=FileWALPointer
[idx=0, fileOff=0, len=0], walSegmentsCleared=0, walSegmentsCovered=[],
markDuration=13ms, pagesWrite=15ms, fsync=11ms, total=39ms]
2019-02-25 10:04:04.082  INFO 1208 --- [oint-thread-#66]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=bd76a704-e125-42e1-b2a8-d285cf15246d, pages=52252,
markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
walSegmentsCovered=[], markDuration=144ms, pagesWrite=1560ms, fsync=123ms,
total=1827ms]
2019-02-25 10:14:27.275  INFO 1208 --- [oint-thread-#66]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=a10f5483-050c-4678-98bd-deca7695e782, pages=368810,
markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
walSegmentsCovered=[], markDuration=323ms, pagesWrite=24669ms, fsync=21ms,
total=25013ms]
2019-02-25 10:24:55.492  INFO 1208 --- [oint-thread-#66]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=bfab59c6-a1a0-4391-92e5-a23b6c416625, pages=382555,
markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
walSegmentsCovered=[], markDuration=225ms, pagesWrite=52912ms, fsync=93ms,
total=53230ms]
2019-02-25 10:35:22.512  INFO 1208 --- [oint-thread-#66]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=feff29e3-db80-4002-82f8-2011e4eca12e, pages=436129,
markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
walSegmentsCovered=[], markDuration=70ms, pagesWrite=80164ms, fsync=7ms,
total=80241ms]
2019-02-25 10:45:34.524  INFO 1208 --- [oint-thread-#66]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=7e3f181b-21d8-4dd1-a018-09711142caf4, pages=384415,
markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
walSegmentsCovered=[], markDuration=56ms, pagesWrite=92186ms, fsync=3ms,
total=92245ms]
2019-02-25 10:55:43.139  INFO 1208 --- [oint-thread-#66]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=a77d48d8-f806-4a48-bfbf-d1afac7d16e3, pages=161643,
markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
walSegmentsCovered=[], markDuration=73ms, pagesWrite=100765ms, fsync=16ms,
total=100854ms]
2019-02-25 11:06:04.437  INFO 1208 --- [oint-thread-#66]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=dbf6a8ab-1c47-4cdd-96f4-fec852dc4fe0, pages=129477,
markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
walSegmentsCovered=[], markDuration=60ms, pagesWrite=121933ms, fsync=154ms,
total=122147ms]
2019-02-25 11:16:08.333  INFO 1208 --- [oint-thread-#66]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=74924992-3dc4-42db-971c-b9030f41d0c5, pages=97511,
markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
walSegmentsCovered=[], markDuration=1676ms, pagesWrite=124336ms, fsync=25ms,
total=126037ms]
2019-02-25 11:18:53.622  INFO 1208 --- [oint-thread-#66]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=7c60efc2-4007-4be2-b3c8-3912d2cbd651, pages=19277,
markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
walSegmentsCovered=[], markDuration=12ms, pagesWrite=47883ms, fsync=344ms,
total=48239ms]


Do you have any other suggestions considering also the behaviour described
above?

Thanks,
Antonio



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Performance degradation in case of high volumes

Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!

It seems that, as time goes, I/O can't catch up with you.

The recommendation here is, probably, to increase checkpoint frequency
value (measured in ms; to do checkpoints less often)
Let's say, set it to 600000 (10 minutes).

The downside here is that in case of crash, node will take more time to
come online.

Regards,
-- 
Ilya Kasnacheev


пт, 22 февр. 2019 г. в 17:37, Antonio Conforti <an...@sia.eu>:

> Hi support,
>
> I'm running a performance test writing 4000 entry per second on a cache:
> 1.      TRANSACTIONAL
> 2.      partitioned
> 3.      with backup 1 (and affinity with exclude neighbors enabled)
> 4.      write synchronization mode FULL_ASYNC
> 5.      indexed on key and value (and enabled to SQL inquiry)
>
> Writes are performed by a client node using a data stream with
> StreamVisitor
> and set autoFlushFrequency 1 sec.
>
> We have configured:
> 1.      failureDetectionTimeout to 120000msec
> 2.      Data region (only 1):
> a.      Persistence enabled
> b.      max size 8 GB
> c.      checkpointPageBufferSize 2 GB
> 3.      WAL mode LOG_ONLY
> 4.      disabled WAL archiving (WAL path and the WAL archive path to the
> same
> value)
> 5.      Pages Writes Throttling enabled
>
>
> After some hour submitting about 20 million entries without problems, the
> client node starts to accuse delays: the queue from the client node Ignite
> reads messages start to grow.
>
> Verifying the logs of server and client node there isn’t any error message
> but from the statistics of WAL  high FSYNC values are observed.
>
> Could you help me to understand why inspite a constant rate and a constant
> consumption of cpu of about 30% only after a certain amount of entry it
> seems the server slow down in term of performance?
>
> May be there is some param to tune that I missed?
>
> Below the configuration used for the simulation:
>
> Total server nodes  8 so distributed:
> HOST1 with 4 nodes server and 1 client node on HDD disk
> HOST2 with 4 nodes on HDD disk
>
>
> Both hosts are machines with 16 cores of 256 GB of memory and HHD disk.
>
> The DataStorageConfiguration for each server node is as follows:
>
>
> <property name="dataStorageConfiguration">
>                 <bean
>
> class="org.apache.ignite.configuration.DataStorageConfiguration">
>
>                         <property name="writeThrottlingEnabled"
> value="true"
> />
>                         <property name="defaultDataRegionConfiguration">
>                                 <bean
>
> class="org.apache.ignite.configuration.DataRegionConfiguration">
>                                         <property name="persistenceEnabled"
> value="true" />
>                                         <property name="maxSize"
> value="#{8L
> * 1024 * 1024 * 1024}"/>
>                                         <property
> name="checkpointPageBufferSize"
>                                                 value="#{2048L * 1024 *
> 1024}" />
>                                 </bean>
>                         </property>
>
>
>
>                         <property name="walMode" value="LOG_ONLY" />
>                         <property name="walPath" value="wal/path" />
>                         <property name="walArchivePath" value="wal/path" />
>                 </bean>
>         </property>
>
>
> JVM option used for start each server node:
>
> -server -Xms4g -Xmx8g -XX:+AlwaysPreTouch -XX:+UseG1GC
> -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
>
>
>
> I report the WAL statistics from log of node 1 :
>
> At Simulation start:
> 2019-02-22 10:19:44.195  INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=c115ead9-643d-45e5-be41-cd7ae5caac14, pages=11891,
> markPos=FileWALPointer [idx=1, fileOff=36517886, len=79426],
> walSegmentsCleared=0, walSegmentsCovered=[0], markDuration=34ms,
> pagesWrite=87ms, fsync=1931ms, total=2052ms]
> 2019-02-22 10:22:44.742  INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=d40c9096-2e10-46ca-ae8e-2a39e242b768, pages=66732,
> markPos=FileWALPointer [idx=7, fileOff=63806638, len=79426],
> walSegmentsCleared=7, walSegmentsCovered=[1 - 6], markDuration=98ms,
> pagesWrite=407ms, fsync=2085ms, total=2590ms]
> 2019-02-22 10:25:44.900  INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=5124f12f-6ed8-4ed3-9644-3c58957600ed, pages=70253,
> markPos=FileWALPointer [idx=14, fileOff=47159207, len=79426],
> walSegmentsCleared=6, walSegmentsCovered=[7 - 13], markDuration=98ms,
> pagesWrite=402ms, fsync=2241ms, total=2741ms]
> 2019-02-22 10:28:47.866  INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=9094d36c-cd79-4d30-9b90-e48de78fa3e6, pages=72524,
> markPos=FileWALPointer [idx=21, fileOff=39728290, len=79426],
> walSegmentsCleared=8, walSegmentsCovered=[14 - 20], markDuration=83ms,
> pagesWrite=365ms, fsync=5255ms, total=5703ms]
> 2019-02-22 10:31:53.635  INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=7132d53a-e2a6-4ac8-b1a8-2621cc39c82b, pages=77471,
> markPos=FileWALPointer [idx=28, fileOff=64681287, len=79426],
> walSegmentsCleared=7, walSegmentsCovered=[21 - 27], markDuration=494ms,
> pagesWrite=748ms, fsync=10136ms, total=11472ms]
>
>
> At end of simulation
>
> 2019-02-22 11:52:36.339  INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=dc8369b9-b100-4dd0-bfaf-9a5b13620072, pages=129942,
> markPos=FileWALPointer [idx=309, fileOff=19048810, len=79426],
> walSegmentsCleared=11, walSegmentsCovered=[298 - 308], markDuration=77ms,
> pagesWrite=797ms, fsync=171049ms, total=171923ms]
> 2019-02-22 11:56:24.001  INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=1fb1ced8-a257-4c6f-be97-a1867ba6692e, pages=133096,
> markPos=FileWALPointer [idx=320, fileOff=13420410, len=79426],
> walSegmentsCleared=11, walSegmentsCovered=[309 - 319], markDuration=1707ms,
> pagesWrite=1537ms, fsync=216332ms, total=219576ms]
> 2019-02-22 12:00:23.052  INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=0d48f6d5-7601-4839-b422-55b228978da5, pages=150587,
> markPos=FileWALPointer [idx=332, fileOff=47800048, len=79426],
> walSegmentsCleared=12, walSegmentsCovered=[320 - 331], markDuration=2275ms,
> pagesWrite=752ms, fsync=236023ms, total=239051ms]
> 2019-02-22 12:04:05.562  INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=b23526c4-5121-48a9-9902-522a5ffb3a28, pages=155805,
> markPos=FileWALPointer [idx=345, fileOff=40020477, len=79426],
> walSegmentsCleared=13, walSegmentsCovered=[332 - 344], markDuration=525ms,
> pagesWrite=1324ms, fsync=220654ms, total=222504ms]
> 2019-02-22 12:07:54.005  INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=1bb2a0c0-7a89-47f2-af9f-90e90d44c14b, pages=149055,
> markPos=FileWALPointer [idx=357, fileOff=51666923, len=79426],
> walSegmentsCleared=12, walSegmentsCovered=[345 - 356], markDuration=995ms,
> pagesWrite=1559ms, fsync=225888ms, total=228442ms]
> 2019-02-22 12:11:49.962  INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=19f14a04-842f-409e-9c6b-afb59193e419, pages=153022,
> markPos=FileWALPointer [idx=370, fileOff=16234647, len=79426],
> walSegmentsCleared=13, walSegmentsCovered=[357 - 369], markDuration=1773ms,
> pagesWrite=1044ms, fsync=233139ms, total=235957ms]
> 2019-02-22 12:15:59.332  INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=c1316fb7-1ecc-4358-bf90-a87772969c03, pages=159668,
> markPos=FileWALPointer [idx=383, fileOff=21979375, len=79426],
> walSegmentsCleared=13, walSegmentsCovered=[370 - 382], markDuration=1249ms,
> pagesWrite=1693ms, fsync=246428ms, total=249370ms]
> 2019-02-22 12:20:05.814  INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=a9e394f2-0011-4f89-8b5a-a6ed77774103, pages=156891,
> markPos=FileWALPointer [idx=396, fileOff=3956799, len=79426],
> walSegmentsCleared=13, walSegmentsCovered=[383 - 395], markDuration=1030ms,
> pagesWrite=1275ms, fsync=244176ms, total=246482ms]
> 2019-02-22 12:24:40.217  INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=0bc71cc5-97c4-4f7d-95c2-430f379eeeb0, pages=148039,
> markPos=FileWALPointer [idx=407, fileOff=57767331, len=79426],
> walSegmentsCleared=11, walSegmentsCovered=[396 - 406], markDuration=323ms,
> pagesWrite=1620ms, fsync=272460ms, total=274403ms]
>
>
> Thanks in advance.
>
> Antonio
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>