You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Antonio Conforti <an...@sia.eu> on 2019/02/22 14:37:41 UTC
Performance degradation in case of high volumes
Hi support,
I'm running a performance test writing 4000 entry per second on a cache:
1. TRANSACTIONAL
2. partitioned
3. with backup 1 (and affinity with exclude neighbors enabled)
4. write synchronization mode FULL_ASYNC
5. indexed on key and value (and enabled to SQL inquiry)
Writes are performed by a client node using a data stream with StreamVisitor
and set autoFlushFrequency 1 sec.
We have configured:
1. failureDetectionTimeout to 120000msec
2. Data region (only 1):
a. Persistence enabled
b. max size 8 GB
c. checkpointPageBufferSize 2 GB
3. WAL mode LOG_ONLY
4. disabled WAL archiving (WAL path and the WAL archive path to the same
value)
5. Pages Writes Throttling enabled
After some hour submitting about 20 million entries without problems, the
client node starts to accuse delays: the queue from the client node Ignite
reads messages start to grow.
Verifying the logs of server and client node there isn’t any error message
but from the statistics of WAL high FSYNC values are observed.
Could you help me to understand why inspite a constant rate and a constant
consumption of cpu of about 30% only after a certain amount of entry it
seems the server slow down in term of performance?
May be there is some param to tune that I missed?
Below the configuration used for the simulation:
Total server nodes 8 so distributed:
HOST1 with 4 nodes server and 1 client node on HDD disk
HOST2 with 4 nodes on HDD disk
Both hosts are machines with 16 cores of 256 GB of memory and HHD disk.
The DataStorageConfiguration for each server node is as follows:
<property name="dataStorageConfiguration">
<bean
class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="writeThrottlingEnabled" value="true"
/>
<property name="defaultDataRegionConfiguration">
<bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="persistenceEnabled"
value="true" />
<property name="maxSize" value="#{8L
* 1024 * 1024 * 1024}"/>
<property
name="checkpointPageBufferSize"
value="#{2048L * 1024 *
1024}" />
</bean>
</property>
<property name="walMode" value="LOG_ONLY" />
<property name="walPath" value="wal/path" />
<property name="walArchivePath" value="wal/path" />
</bean>
</property>
JVM option used for start each server node:
-server -Xms4g -Xmx8g -XX:+AlwaysPreTouch -XX:+UseG1GC
-XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
I report the WAL statistics from log of node 1 :
At Simulation start:
2019-02-22 10:19:44.195 INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=c115ead9-643d-45e5-be41-cd7ae5caac14, pages=11891,
markPos=FileWALPointer [idx=1, fileOff=36517886, len=79426],
walSegmentsCleared=0, walSegmentsCovered=[0], markDuration=34ms,
pagesWrite=87ms, fsync=1931ms, total=2052ms]
2019-02-22 10:22:44.742 INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=d40c9096-2e10-46ca-ae8e-2a39e242b768, pages=66732,
markPos=FileWALPointer [idx=7, fileOff=63806638, len=79426],
walSegmentsCleared=7, walSegmentsCovered=[1 - 6], markDuration=98ms,
pagesWrite=407ms, fsync=2085ms, total=2590ms]
2019-02-22 10:25:44.900 INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=5124f12f-6ed8-4ed3-9644-3c58957600ed, pages=70253,
markPos=FileWALPointer [idx=14, fileOff=47159207, len=79426],
walSegmentsCleared=6, walSegmentsCovered=[7 - 13], markDuration=98ms,
pagesWrite=402ms, fsync=2241ms, total=2741ms]
2019-02-22 10:28:47.866 INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=9094d36c-cd79-4d30-9b90-e48de78fa3e6, pages=72524,
markPos=FileWALPointer [idx=21, fileOff=39728290, len=79426],
walSegmentsCleared=8, walSegmentsCovered=[14 - 20], markDuration=83ms,
pagesWrite=365ms, fsync=5255ms, total=5703ms]
2019-02-22 10:31:53.635 INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=7132d53a-e2a6-4ac8-b1a8-2621cc39c82b, pages=77471,
markPos=FileWALPointer [idx=28, fileOff=64681287, len=79426],
walSegmentsCleared=7, walSegmentsCovered=[21 - 27], markDuration=494ms,
pagesWrite=748ms, fsync=10136ms, total=11472ms]
At end of simulation
2019-02-22 11:52:36.339 INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=dc8369b9-b100-4dd0-bfaf-9a5b13620072, pages=129942,
markPos=FileWALPointer [idx=309, fileOff=19048810, len=79426],
walSegmentsCleared=11, walSegmentsCovered=[298 - 308], markDuration=77ms,
pagesWrite=797ms, fsync=171049ms, total=171923ms]
2019-02-22 11:56:24.001 INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=1fb1ced8-a257-4c6f-be97-a1867ba6692e, pages=133096,
markPos=FileWALPointer [idx=320, fileOff=13420410, len=79426],
walSegmentsCleared=11, walSegmentsCovered=[309 - 319], markDuration=1707ms,
pagesWrite=1537ms, fsync=216332ms, total=219576ms]
2019-02-22 12:00:23.052 INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=0d48f6d5-7601-4839-b422-55b228978da5, pages=150587,
markPos=FileWALPointer [idx=332, fileOff=47800048, len=79426],
walSegmentsCleared=12, walSegmentsCovered=[320 - 331], markDuration=2275ms,
pagesWrite=752ms, fsync=236023ms, total=239051ms]
2019-02-22 12:04:05.562 INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=b23526c4-5121-48a9-9902-522a5ffb3a28, pages=155805,
markPos=FileWALPointer [idx=345, fileOff=40020477, len=79426],
walSegmentsCleared=13, walSegmentsCovered=[332 - 344], markDuration=525ms,
pagesWrite=1324ms, fsync=220654ms, total=222504ms]
2019-02-22 12:07:54.005 INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=1bb2a0c0-7a89-47f2-af9f-90e90d44c14b, pages=149055,
markPos=FileWALPointer [idx=357, fileOff=51666923, len=79426],
walSegmentsCleared=12, walSegmentsCovered=[345 - 356], markDuration=995ms,
pagesWrite=1559ms, fsync=225888ms, total=228442ms]
2019-02-22 12:11:49.962 INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=19f14a04-842f-409e-9c6b-afb59193e419, pages=153022,
markPos=FileWALPointer [idx=370, fileOff=16234647, len=79426],
walSegmentsCleared=13, walSegmentsCovered=[357 - 369], markDuration=1773ms,
pagesWrite=1044ms, fsync=233139ms, total=235957ms]
2019-02-22 12:15:59.332 INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=c1316fb7-1ecc-4358-bf90-a87772969c03, pages=159668,
markPos=FileWALPointer [idx=383, fileOff=21979375, len=79426],
walSegmentsCleared=13, walSegmentsCovered=[370 - 382], markDuration=1249ms,
pagesWrite=1693ms, fsync=246428ms, total=249370ms]
2019-02-22 12:20:05.814 INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=a9e394f2-0011-4f89-8b5a-a6ed77774103, pages=156891,
markPos=FileWALPointer [idx=396, fileOff=3956799, len=79426],
walSegmentsCleared=13, walSegmentsCovered=[383 - 395], markDuration=1030ms,
pagesWrite=1275ms, fsync=244176ms, total=246482ms]
2019-02-22 12:24:40.217 INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=0bc71cc5-97c4-4f7d-95c2-430f379eeeb0, pages=148039,
markPos=FileWALPointer [idx=407, fileOff=57767331, len=79426],
walSegmentsCleared=11, walSegmentsCovered=[396 - 406], markDuration=323ms,
pagesWrite=1620ms, fsync=272460ms, total=274403ms]
Thanks in advance.
Antonio
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Performance degradation in case of high volumes
Posted by Justin Ji <bi...@foxmail.com>.
Ilya -
I use ignite 2.6.0, does not have IGNITE_MAX_INDEX_PAYLOAD_SIZE system
property.
But our index field has a fixed length:25 characters, so where can I find
the algorithm to calculate the 'index inline size'.
Looking forward to your reply.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Performance degradation in case of high volumes
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
I'm afraid that at this point even indexes do not fit in RAM so every
insert needs to get index pages from disk, change them and put them back to
make room for newer ones. Please consider having larger DataRegions.
Regards,
--
Ilya Kasnacheev
ср, 6 мар. 2019 г. в 16:26, Antonio Conforti <an...@sia.eu>:
> Hello Ilya,
>
>
> thanks for reply.
>
> For answer to your questions today i runned again a test that i stopped at
> about 22 millions of entries:
>
> a) the approximate hardware and data region config used in the test:
>
> 2 hosts each with:
>
> Processor total : 2
>
> Processor: 0
> Name : Intel Xeon
> Speed : 3300 MHz
> Bus : 100 MHz
> Core : 8
> Thread : 16
> Socket : 1
> Level2 Cache : 2048 KBytes
> Level3 Cache : 25600 KBytes
>
> Processor: 1
> Name : Intel Xeon
> Stepping : 4
> Speed : 3300 MHz
> Bus : 100 MHz
> Core : 8
> Thread : 16
> Socket : 2
> Level2 Cache : 2048 KBytes
> Level3 Cache : 25600 KBytes
>
> MEMORY:
> Memory installed : 196608 MBytes
>
> # getconf PAGESIZE: 4096
>
> vm.swappiness = 5
>
>
>
>
>
> Disk:
>
> Cache Board Present: True
> Cache Status: OK
> Cache Ratio: 25% Read / 75% Write
> Drive Write Cache: Disabled
> Total Cache Size: 512 MB
> Total Cache Memory Available: 304 MB
>
> Logical Drive: 1
> Size: 279.4 GB
> Fault Tolerance: RAID 1
> Heads: 255
> Sectors Per Track: 32
> Cylinders: 65535
> Strip Size: 256 KB
> Full Stripe Size: 256 KB
> Status: OK
> Caching: Enabled
> Rotational Speed: 15000
> PHY Transfer Rate: 6.0Gbps
> PHY Transfer Rate: 6.0Gbps
> type ext4
> Block size: 4096
>
>
> From the attached xml config the default data storage configuration is:
>
>
> datafeed-persistent-store.xml
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t2315/datafeed-persistent-store.xml>
>
>
> <property name="maxSize" value="#{8L * 1024 * 1024 * 1024}"/>
>
> <property name="checkpointPageBufferSize" value="#{2048L * 1024 * 1024}" />
>
>
> About the memory below the top command for server and client nodes before
> start test:
>
>
>
>
> HOST_1
>
>
>
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t2315/HOST1_top_preTest.png>
>
>
> HOST_2
>
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t2315/HOST2_top_preTest.png>
>
>
> and below the top command for server and client nodes when test is stopped:
>
> HOST_1
>
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t2315/HOST1_top_postTest.png>
>
>
> HOST_2
>
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t2315/HOST2_top_postTest.png>
>
>
>
> b) approximate size of data is 656 bytes
> c) The size of db/ directory at about 22 millions is
>
> For HOST_1:
> 6.7G Work_1/
> 5.3G Work_3/
> 6.3G Work_5/
> 7.2G Work_7/
>
>
>
> For HOST_2:
> 6.5G Work_2
> 6.5G Work_4
> 6.6G Work_6
> 5.9G Work_8
>
> Thanks,
>
> Antonio
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Performance degradation in case of high volumes
Posted by Antonio Conforti <an...@sia.eu>.
Hello Ilya,
thanks for reply.
For answer to your questions today i runned again a test that i stopped at
about 22 millions of entries:
a) the approximate hardware and data region config used in the test:
2 hosts each with:
Processor total : 2
Processor: 0
Name : Intel Xeon
Speed : 3300 MHz
Bus : 100 MHz
Core : 8
Thread : 16
Socket : 1
Level2 Cache : 2048 KBytes
Level3 Cache : 25600 KBytes
Processor: 1
Name : Intel Xeon
Stepping : 4
Speed : 3300 MHz
Bus : 100 MHz
Core : 8
Thread : 16
Socket : 2
Level2 Cache : 2048 KBytes
Level3 Cache : 25600 KBytes
MEMORY:
Memory installed : 196608 MBytes
# getconf PAGESIZE: 4096
vm.swappiness = 5
Disk:
Cache Board Present: True
Cache Status: OK
Cache Ratio: 25% Read / 75% Write
Drive Write Cache: Disabled
Total Cache Size: 512 MB
Total Cache Memory Available: 304 MB
Logical Drive: 1
Size: 279.4 GB
Fault Tolerance: RAID 1
Heads: 255
Sectors Per Track: 32
Cylinders: 65535
Strip Size: 256 KB
Full Stripe Size: 256 KB
Status: OK
Caching: Enabled
Rotational Speed: 15000
PHY Transfer Rate: 6.0Gbps
PHY Transfer Rate: 6.0Gbps
type ext4
Block size: 4096
From the attached xml config the default data storage configuration is:
datafeed-persistent-store.xml
<http://apache-ignite-users.70518.x6.nabble.com/file/t2315/datafeed-persistent-store.xml>
<property name="maxSize" value="#{8L * 1024 * 1024 * 1024}"/>
<property name="checkpointPageBufferSize" value="#{2048L * 1024 * 1024}" />
About the memory below the top command for server and client nodes before
start test:
HOST_1
<http://apache-ignite-users.70518.x6.nabble.com/file/t2315/HOST1_top_preTest.png>
HOST_2
<http://apache-ignite-users.70518.x6.nabble.com/file/t2315/HOST2_top_preTest.png>
and below the top command for server and client nodes when test is stopped:
HOST_1
<http://apache-ignite-users.70518.x6.nabble.com/file/t2315/HOST1_top_postTest.png>
HOST_2
<http://apache-ignite-users.70518.x6.nabble.com/file/t2315/HOST2_top_postTest.png>
b) approximate size of data is 656 bytes
c) The size of db/ directory at about 22 millions is
For HOST_1:
6.7G Work_1/
5.3G Work_3/
6.3G Work_5/
7.2G Work_7/
For HOST_2:
6.5G Work_2
6.5G Work_4
6.6G Work_6
5.9G Work_8
Thanks,
Antonio
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Performance degradation in case of high volumes
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Can you please share a) approximate hardware and data region config that
you are using, and b) approximate size of your data.
How large is 18 million entries? What's the size of db/ directory at this
point?
Regards,
--
Ilya Kasnacheev
пн, 4 мар. 2019 г. в 17:36, Antonio Conforti <an...@sia.eu>:
> Hello Ilya,
>
> not running out of checkpoint buffers is certainly good news..
>
> I run today another test to collect the information you asked for, with the
> same configuration of the test executed in date 2019/03/01 and previously
> discussed in this thread.
> Below the details of entries per node :
>
> Cache 'DF_CMF_QUOTE(@c0)':
> +------------------------------------------------------------------+
> | Name(@) | DF_CMF_QUOTE(@c0) |
> | Total entries (Heap / Off-heap) | 24050018 (0 / 24050018) |
> | Nodes | 8 |
> | Total size Min/Avg/Max | 2419854 / 3006252.25 / 3426152 |
> | Heap size Min/Avg/Max | 0 / 0.00 / 0 |
> | Off-heap size Min/Avg/Max | 2419854 / 3006252.25 / 3426152 |
> +------------------------------------------------------------------+
>
> Nodes for: DF_CMF_QUOTE(@c0)
>
> +==============================================================================================================================+
> | Node ID8(@), IP | CPUs | Heap Used | CPU Load | Up Time
>
> | Size (Primary / Backup) | Hi/Mi/Rd/Wr |
>
> +==============================================================================================================================+
> | 269DFE65(@n7), HOST_1 | 16 | 2.67 % | 0.03 % |
> 02:25:29.411
> | Total: 3147250 (3147250 / 0) | Hi: 0 |
> | | | | |
>
> | Heap: 0 (0 / <n/a>) | Mi: 0 |
> | | | | |
>
> | Off-Heap: 3147250 (3147250 / 0) | Rd: 0 |
> | | | | |
>
> | Off-Heap Memory: <n/a> | Wr: 0 |
>
> +-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
> | FF347987(@n6), HOST_1 | 16 | 23.93 % | 0.03 % |
> 02:25:17.106
> | Total: 2820298 (2820298 / 0) | Hi: 0 |
> | | | | |
>
> | Heap: 0 (0 / <n/a>) | Mi: 0 |
> | | | | |
>
> | Off-Heap: 2820298 (2820298 / 0) | Rd: 0 |
> | | | | |
>
> | Off-Heap Memory: <n/a> | Wr: 0 |
>
> +-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
> | 5DF4A6EE(@n4), HOST_1 | 16 | 12.12 % | 0.00 % |
> 02:25:34.673
> | Total: 3077901 (3077901 / 0) | Hi: 0 |
> | | | | |
>
> | Heap: 0 (0 / <n/a>) | Mi: 0 |
> | | | | |
>
> | Off-Heap: 3077901 (3077901 / 0) | Rd: 0 |
> | | | | |
>
> | Off-Heap Memory: <n/a> | Wr: 0 |
>
> +-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
> | 51133869(@n5), HOST_1 | 16 | 2.44 % | 0.00 % |
> 02:25:23.450
> | Total: 3146224 (3146224 / 0) | Hi: 0 |
> | | | | |
>
> | Heap: 0 (0 / <n/a>) | Mi: 0 |
> | | | | |
>
> | Off-Heap: 3146224 (3146224 / 0) | Rd: 0 |
> | | | | |
>
> | Off-Heap Memory: <n/a> | Wr: 0 |
>
> +-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
> | E4CC3158(@n0), HOST_2 | 16 | 5.15 % | 0.00 % |
> 02:26:51.374
> | Total: 3073950 (3073950 / 0) | Hi: 0 |
> | | | | |
>
> | Heap: 0 (0 / <n/a>) | Mi: 0 |
> | | | | |
>
> | Off-Heap: 3073950 (3073950 / 0) | Rd: 0 |
> | | | | |
>
> | Off-Heap Memory: <n/a> | Wr: 0 |
>
> +-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
> | EB897A74(@n2), HOST_2 | 16 | 8.85 % | 0.03 % |
> 02:26:36.007
> | Total: 2938389 (2938389 / 0) | Hi: 0 |
> | | | | |
>
> | Heap: 0 (0 / <n/a>) | Mi: 0 |
> | | | | |
>
> | Off-Heap: 2938389 (2938389 / 0) | Rd: 0 |
> | | | | |
>
> | Off-Heap Memory: <n/a> | Wr: 0 |
>
> +-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
> | 38404C41(@n3), HOST_2 | 16 | 5.49 % | 0.03 % |
> 02:26:29.103
> | Total: 3426152 (3426152 / 0) | Hi: 0 |
> | | | | |
>
> | Heap: 0 (0 / <n/a>) | Mi: 0 |
> | | | | |
>
> | Off-Heap: 3426152 (3426152 / 0) | Rd: 0 |
> | | | | |
>
> | Off-Heap Memory: <n/a> | Wr: 0 |
>
> +-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
> | 570D0880(@n1), HOST_2 | 16 | 26.10 % | 0.03 % |
> 02:26:41.175
> | Total: 2419854 (2419854 / 0) | Hi: 0 |
> | | | | |
>
> | Heap: 0 (0 / <n/a>) | Mi: 0 |
> | | | | |
>
> | Off-Heap: 2419854 (2419854 / 0) | Rd: 0 |
> | | | | |
>
> | Off-Heap Memory: <n/a> | Wr: 0 |
>
> +------------------------------------------------------------------------------------------------------------------------------+
>
> Some comments about the test:
>
> 1) When I start the test with the cache empty, I begin to observe the
> degradation of performance after about 1 hour and 15 minutes and,
> particularly, when I'm around a total of 18 million entries.
> Before that number of entries everything seems to be fine.
>
> Below the statistics at the end of the checkpoint read in log file for
> node 7:
>
> Run A) for CONSISTENDID 7
> Stop time checkPoint Num Pages Elapsed Checkpoint ms
> 09:25:49.016 18353
> 449
> 09:28:49.941 59593
> 1368
> 09:31:50.057 62797
> 1480
> 09:34:50.234 65509
> 1655
> 09:37:50.489 69339
> 1902
> 09:40:56.005 72438
> 7410
> 09:44:03.367 74923
> 14767
> 09:47:07.403 78554
> 18799
> 09:50:13.981 81913
> 25372
> 09:53:24.503 85730
> 35889
> 09:56:27.091 88078
> 38468
> 09:59:31.535 90275
> 42913
> 10:02:35.038 92802
> 46409
> 10:05:40.214 95503
> 51579
> 10:08:49.639 97856
> 60996
> 10:11:58.841 101746
> 70203
> 10:15:02.559 104501
> 73913
> 10:18:14.915 106753
> 86266
> 10:21:17.593 108685
> 88942
> 10:24:25.780 111062
> 97126
> 10:27:33.036 112904
> 104376
> 10:30:35.121 113809
> 106461
> 10:33:40.040 115920
> 111374
> 10:39:46.645 118431
> 117973
> 10:42:54.245 120306
> 125568
> 10:45:56.420 121766
> 127737
> 10:49:02.531 123532
> 133845
> 10:52:04.470 125685
> 135785
> 10:55:12.588 127601
> 143895
> 10:58:16.771 129164
> 148075
> 11:00:59.904 102594
> 131206
>
>
>
> 2) If I stop the test when I begin to observe the degradation of
> performance
> described in point 1) at about 11:00 AM and I wait for a while, precisely
> since in the log of my servers nodes I read "Skipping checkpoint (no pages
> were modified)" in order to be sure that no pending entries are in
> processing and then I run again the test submitting 4000 entry per second,
> I
> observe in a short time a degradation of the performances.
>
> Below the statistic at the end of the checkpoint read in log file fornode
> 7:
>
> Run B) for CONSISTENDID 7
> Stop time checkPoint Num Pages Elapsed Checkpoint ms
> 11:23:09.120 24743
> 20386
> 11:28:58.436 131858
> 189695
> 11:32:45.824 137487
> 227388
> 11:33:53.565 53774
> 67741
>
>
> Please note also as highlighted above in table of point 1) the progressive
> grow of pages and consequent elapsed in the execution times of checkpoints.
> A note also inherent the pages managed by checkpoint in run b) described in
> point 2) compared to pages managed by checkpoint of run A) described in
> point 1):
> the number of pages managed at the start of run B) are quite the same as
> those at the end of run A) and not, as I expected, similar to those at the
> start of the initial run A).
>
> The observations above might suggest to tune some other configuration
> parameter?
>
> What may be, in your opinion, the cause of this performance degradation
> when I reach about 18 milions of entries?
> What system resource may be exhausted ?
>
> Can you also tell me in what log do you observe the different
> configurations
> ? The configuration file used for all nodes seems to me the same but
> probably I'm missing something ...
>
> In attach the logs about the run test 2019/03/01
> log_ignite_190304_HOST1.gz
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t2315/log_ignite_190304_HOST1.gz>
>
> log_ignite_190304_HOST2.gz
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t2315/log_ignite_190304_HOST2.gz>
>
>
> Thanks,
> Antonio
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Performance degradation in case of high volumes
Posted by Antonio Conforti <an...@sia.eu>.
Hello Ilya,
not running out of checkpoint buffers is certainly good news..
I run today another test to collect the information you asked for, with the
same configuration of the test executed in date 2019/03/01 and previously
discussed in this thread.
Below the details of entries per node :
Cache 'DF_CMF_QUOTE(@c0)':
+------------------------------------------------------------------+
| Name(@) | DF_CMF_QUOTE(@c0) |
| Total entries (Heap / Off-heap) | 24050018 (0 / 24050018) |
| Nodes | 8 |
| Total size Min/Avg/Max | 2419854 / 3006252.25 / 3426152 |
| Heap size Min/Avg/Max | 0 / 0.00 / 0 |
| Off-heap size Min/Avg/Max | 2419854 / 3006252.25 / 3426152 |
+------------------------------------------------------------------+
Nodes for: DF_CMF_QUOTE(@c0)
+==============================================================================================================================+
| Node ID8(@), IP | CPUs | Heap Used | CPU Load | Up Time
| Size (Primary / Backup) | Hi/Mi/Rd/Wr |
+==============================================================================================================================+
| 269DFE65(@n7), HOST_1 | 16 | 2.67 % | 0.03 % | 02:25:29.411
| Total: 3147250 (3147250 / 0) | Hi: 0 |
| | | | |
| Heap: 0 (0 / <n/a>) | Mi: 0 |
| | | | |
| Off-Heap: 3147250 (3147250 / 0) | Rd: 0 |
| | | | |
| Off-Heap Memory: <n/a> | Wr: 0 |
+-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
| FF347987(@n6), HOST_1 | 16 | 23.93 % | 0.03 % | 02:25:17.106
| Total: 2820298 (2820298 / 0) | Hi: 0 |
| | | | |
| Heap: 0 (0 / <n/a>) | Mi: 0 |
| | | | |
| Off-Heap: 2820298 (2820298 / 0) | Rd: 0 |
| | | | |
| Off-Heap Memory: <n/a> | Wr: 0 |
+-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
| 5DF4A6EE(@n4), HOST_1 | 16 | 12.12 % | 0.00 % | 02:25:34.673
| Total: 3077901 (3077901 / 0) | Hi: 0 |
| | | | |
| Heap: 0 (0 / <n/a>) | Mi: 0 |
| | | | |
| Off-Heap: 3077901 (3077901 / 0) | Rd: 0 |
| | | | |
| Off-Heap Memory: <n/a> | Wr: 0 |
+-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
| 51133869(@n5), HOST_1 | 16 | 2.44 % | 0.00 % | 02:25:23.450
| Total: 3146224 (3146224 / 0) | Hi: 0 |
| | | | |
| Heap: 0 (0 / <n/a>) | Mi: 0 |
| | | | |
| Off-Heap: 3146224 (3146224 / 0) | Rd: 0 |
| | | | |
| Off-Heap Memory: <n/a> | Wr: 0 |
+-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
| E4CC3158(@n0), HOST_2 | 16 | 5.15 % | 0.00 % | 02:26:51.374
| Total: 3073950 (3073950 / 0) | Hi: 0 |
| | | | |
| Heap: 0 (0 / <n/a>) | Mi: 0 |
| | | | |
| Off-Heap: 3073950 (3073950 / 0) | Rd: 0 |
| | | | |
| Off-Heap Memory: <n/a> | Wr: 0 |
+-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
| EB897A74(@n2), HOST_2 | 16 | 8.85 % | 0.03 % | 02:26:36.007
| Total: 2938389 (2938389 / 0) | Hi: 0 |
| | | | |
| Heap: 0 (0 / <n/a>) | Mi: 0 |
| | | | |
| Off-Heap: 2938389 (2938389 / 0) | Rd: 0 |
| | | | |
| Off-Heap Memory: <n/a> | Wr: 0 |
+-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
| 38404C41(@n3), HOST_2 | 16 | 5.49 % | 0.03 % | 02:26:29.103
| Total: 3426152 (3426152 / 0) | Hi: 0 |
| | | | |
| Heap: 0 (0 / <n/a>) | Mi: 0 |
| | | | |
| Off-Heap: 3426152 (3426152 / 0) | Rd: 0 |
| | | | |
| Off-Heap Memory: <n/a> | Wr: 0 |
+-------------------------------+------+-----------+----------+--------------+-----------------------------------+-------------+
| 570D0880(@n1), HOST_2 | 16 | 26.10 % | 0.03 % | 02:26:41.175
| Total: 2419854 (2419854 / 0) | Hi: 0 |
| | | | |
| Heap: 0 (0 / <n/a>) | Mi: 0 |
| | | | |
| Off-Heap: 2419854 (2419854 / 0) | Rd: 0 |
| | | | |
| Off-Heap Memory: <n/a> | Wr: 0 |
+------------------------------------------------------------------------------------------------------------------------------+
Some comments about the test:
1) When I start the test with the cache empty, I begin to observe the
degradation of performance after about 1 hour and 15 minutes and,
particularly, when I'm around a total of 18 million entries.
Before that number of entries everything seems to be fine.
Below the statistics at the end of the checkpoint read in log file for
node 7:
Run A) for CONSISTENDID 7
Stop time checkPoint Num Pages Elapsed Checkpoint ms
09:25:49.016 18353
449
09:28:49.941 59593
1368
09:31:50.057 62797
1480
09:34:50.234 65509
1655
09:37:50.489 69339
1902
09:40:56.005 72438
7410
09:44:03.367 74923
14767
09:47:07.403 78554
18799
09:50:13.981 81913
25372
09:53:24.503 85730
35889
09:56:27.091 88078
38468
09:59:31.535 90275
42913
10:02:35.038 92802
46409
10:05:40.214 95503
51579
10:08:49.639 97856
60996
10:11:58.841 101746
70203
10:15:02.559 104501
73913
10:18:14.915 106753
86266
10:21:17.593 108685
88942
10:24:25.780 111062
97126
10:27:33.036 112904
104376
10:30:35.121 113809
106461
10:33:40.040 115920
111374
10:39:46.645 118431
117973
10:42:54.245 120306
125568
10:45:56.420 121766
127737
10:49:02.531 123532
133845
10:52:04.470 125685
135785
10:55:12.588 127601
143895
10:58:16.771 129164
148075
11:00:59.904 102594
131206
2) If I stop the test when I begin to observe the degradation of performance
described in point 1) at about 11:00 AM and I wait for a while, precisely
since in the log of my servers nodes I read "Skipping checkpoint (no pages
were modified)" in order to be sure that no pending entries are in
processing and then I run again the test submitting 4000 entry per second, I
observe in a short time a degradation of the performances.
Below the statistic at the end of the checkpoint read in log file fornode 7:
Run B) for CONSISTENDID 7
Stop time checkPoint Num Pages Elapsed Checkpoint ms
11:23:09.120 24743
20386
11:28:58.436 131858
189695
11:32:45.824 137487
227388
11:33:53.565 53774
67741
Please note also as highlighted above in table of point 1) the progressive
grow of pages and consequent elapsed in the execution times of checkpoints.
A note also inherent the pages managed by checkpoint in run b) described in
point 2) compared to pages managed by checkpoint of run A) described in
point 1):
the number of pages managed at the start of run B) are quite the same as
those at the end of run A) and not, as I expected, similar to those at the
start of the initial run A).
The observations above might suggest to tune some other configuration
parameter?
What may be, in your opinion, the cause of this performance degradation
when I reach about 18 milions of entries?
What system resource may be exhausted ?
Can you also tell me in what log do you observe the different configurations
? The configuration file used for all nodes seems to me the same but
probably I'm missing something ...
In attach the logs about the run test 2019/03/01
log_ignite_190304_HOST1.gz
<http://apache-ignite-users.70518.x6.nabble.com/file/t2315/log_ignite_190304_HOST1.gz>
log_ignite_190304_HOST2.gz
<http://apache-ignite-users.70518.x6.nabble.com/file/t2315/log_ignite_190304_HOST2.gz>
Thanks,
Antonio
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Performance degradation in case of high volumes
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello Antonio!
I canonly observe 'timeout' checkpoints which is good news - you are not
running out of checkpoint buffer.
Otherwise, maybe you are hitting actual performance limit, i.e., your
system is saturated for good at this point. What is total amount of data
per node at this time? What's the size of your cache entry?
I have also noted that your 2 nodes have wildly different config. The first
one has around 39G of data region, the second one has 8G.
Regards,
--
Ilya Kasnacheev
пт, 1 мар. 2019 г. в 17:19, Antonio Conforti <an...@sia.eu>:
> Hello Ilya.
>
> I ran again the test from scratch with fixed rate at 4000 msg/sec. with the
> environment variable IGNITE_MAX_INDEX_PAYLOAD_SIZE=66 and Cache:
>
> 1) PARTITIONED
> 2) TRANSACTIONAL
> 3) persistence enabled
> 4) backup=0
> 5) indexes on key and value
> 6) Data region 8 GB
> 7) Checkpoint buffer size 2 GB
> 8) WAL mode LOG_ONLY
> 9) WAL archive disabled
> 10)Pages Writes Throttling enabled)
>
> During the test I could observe the total checkpoint elapsed grew up and
> the
> platform processed all the messages without queueing. The throttling was
> logged in every checkpoint.
> After about 18 milions of entries the performance slowdown and queues
> formed. The total checkpoint elapsed exceeded the checkpoint timeout
> (default).
> A lot of "Critical system error detected" were logged and the platform
> never
> recovered the performance.
> I observed also the data region was filled after the perfomance slowdown
> and
> the checkpoint elapsed exceeded the timeout.
>
> In attach the logs and configuration files.
>
> log_ignite_190301_HOST1.gz
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t2315/log_ignite_190301_HOST1.gz>
>
> log_ignite_190301_HOST2.gz
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t2315/log_ignite_190301_HOST2.gz>
>
>
> Waiting your suggestions let me know if may be usueful a reproducer project
> for the slow down of performance that i observed
>
> Thanks.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Performance degradation in case of high volumes
Posted by Antonio Conforti <an...@sia.eu>.
Hello Ilya.
I ran again the test from scratch with fixed rate at 4000 msg/sec. with the
environment variable IGNITE_MAX_INDEX_PAYLOAD_SIZE=66 and Cache:
1) PARTITIONED
2) TRANSACTIONAL
3) persistence enabled
4) backup=0
5) indexes on key and value
6) Data region 8 GB
7) Checkpoint buffer size 2 GB
8) WAL mode LOG_ONLY
9) WAL archive disabled
10)Pages Writes Throttling enabled)
During the test I could observe the total checkpoint elapsed grew up and the
platform processed all the messages without queueing. The throttling was
logged in every checkpoint.
After about 18 milions of entries the performance slowdown and queues
formed. The total checkpoint elapsed exceeded the checkpoint timeout
(default).
A lot of "Critical system error detected" were logged and the platform never
recovered the performance.
I observed also the data region was filled after the perfomance slowdown and
the checkpoint elapsed exceeded the timeout.
In attach the logs and configuration files.
log_ignite_190301_HOST1.gz
<http://apache-ignite-users.70518.x6.nabble.com/file/t2315/log_ignite_190301_HOST1.gz>
log_ignite_190301_HOST2.gz
<http://apache-ignite-users.70518.x6.nabble.com/file/t2315/log_ignite_190301_HOST2.gz>
Waiting your suggestions let me know if may be usueful a reproducer project
for the slow down of performance that i observed
Thanks.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Performance degradation in case of high volumes
Posted by Antonio Conforti <an...@sia.eu>.
Hello Ilya,
you are right from 65681 to 82206 and the corresponding duration of
checkpoint from 1,6 sec to 9,6 sec it is not catastrophic. I just wanted to
highlight a noticeable increase in dirty pages in a short time despite the
constant rate, suggesting that in the long run the performance would be
degraded. Unfortunately I could not finish the test but right now I'm doing
a complete test: 4000 entries per second trying to get to 60 million
entries.
I'll post you results as soon as possible.
Thanks again for your collaboration!
Antonio
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Performance degradation in case of high volumes
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello?
What's delay of checkpoint?
I fail to see how 82206 is catastrophically worse than 65681. To me it
sounds like sensible increase.
Regards,
--
Ilya Kasnacheev
чт, 28 февр. 2019 г. в 19:01, Antonio Conforti <an...@sia.eu>:
> Hello Ilya,
>
>
> if I well interpret the log for example of node with consitentid 7 what I
> see is a constant increment of dirty pages:
>
> 2019-02-27 17:04:42.206 INFO 41153 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=b3d5b34c-60d3-4c39-af0f-5de6d41686f3, pages=65681,
> markPos=FileWALPointer [idx=16, fileOff=24231459, len=75645],
> walSegmentsCleared=6, walSegmentsCovered=[10 - 15], markDuration=74ms,
> pagesWrite=1429ms, fsync=101ms, total=1604ms]
> 2019-02-27 17:07:42.244 INFO 41153 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=83cc0c0a-9427-465d-a290-f8a56c366528, pages=69139,
> markPos=FileWALPointer [idx=22, fileOff=41402244, len=75645],
> walSegmentsCleared=6, walSegmentsCovered=[16 - 21], markDuration=26ms,
> pagesWrite=1521ms, fsync=101ms, total=1648ms]
> 2019-02-27 17:10:42.553 INFO 41153 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=4fe0396f-5ece-4016-a269-119d5ecd8d16, pages=72896,
> markPos=FileWALPointer [idx=29, fileOff=6850845, len=75645],
> walSegmentsCleared=12, walSegmentsCovered=[22 - 28], markDuration=28ms,
> pagesWrite=1713ms, fsync=207ms, total=1948ms]
> 2019-02-27 17:13:45.214 INFO 41153 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=a309e6a2-865d-439e-ab6d-11ef9641c8b0, pages=75381,
> markPos=FileWALPointer [idx=35, fileOff=48279077, len=75645],
> walSegmentsCleared=7, walSegmentsCovered=[29 - 34], markDuration=44ms,
> pagesWrite=4446ms, fsync=112ms, total=4602ms]
> 2019-02-27 17:16:42.728 INFO 41153 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=0299f7d3-8ed0-4f0b-8499-14e46d3bef13, pages=78944,
> markPos=FileWALPointer [idx=42, fileOff=39209416, len=75645],
> walSegmentsCleared=6, walSegmentsCovered=[35 - 41], markDuration=28ms,
> pagesWrite=1989ms, fsync=95ms, total=2112ms]
> 2019-02-27 17:19:40.829 INFO 41153 --- [r-stripe-10-#27]
> o.a.i.i.p.c.p.pagemem.PageMemoryImpl : Throttling is applied to page
> modifications [percentOfPartTime=0.31, markDirty=22777 pages/sec,
> checkpointWrite=11469 pages/sec, estIdealMarkDirty=226519 pages/sec,
> curDirty=0.00, maxDirty=0.07, avgParkTime=122057 ns, pages: (total=82206,
> evicted=0, written=10035, synced=0, cpBufUsed=27, cpBufTotal=518215)]
> 2019-02-27 17:19:50.524 INFO 41153 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=f323c2fd-d358-469e-8e12-d2ae1391a546, pages=82206,
> markPos=FileWALPointer [idx=49, fileOff=42053202, len=75645],
> walSegmentsCleared=8, walSegmentsCovered=[42 - 48], markDuration=27ms,
> pagesWrite=9749ms, fsync=123ms, total=9899ms]
> 2019-02-27 17:22:40.885 INFO 41153 --- [r-stripe-11-#28]
> o.a.i.i.p.c.p.pagemem.PageMemoryImpl : Throttling is applied to page
> modifications [percentOfPartTime=0.23, markDirty=14449 pages/sec,
> checkpointWrite=6729 pages/sec, estIdealMarkDirty=166922 pages/sec,
> curDirty=0.00, maxDirty=0.09, avgParkTime=108997 ns, pages: (total=68092,
> evicted=0, written=13266, synced=0, cpBufUsed=12, cpBufTotal=518215)]
>
>
> I try to explain better:
>
> I see at every finished checkpoint a constant increease of dirty pages and
> a
> consequent duration increase of checkpoint:
>
> 1) at 17:04:42.206 dirty pages are 65681 with a delay of checkpoint of
> 1604ms
> 2) at 17:19:50.524 dirty pages are 82206 with a delay of checkpoint of
> 9899ms
>
>
> So in only 15 minutes with constant rate of 4000 entry/sec I observe a
> increment of delay of about 9 times respect to start of simulation and the
> consequent apllication of throttling how reported in the log at 17:19:40.
>
> I expected a constant number of dirty page and a consequent constant delay
> of checkpoint.
>
> Do you have any suggestion about that?
>
> Thanks a lot.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Performance degradation in case of high volumes
Posted by Antonio Conforti <an...@sia.eu>.
Hello Ilya,
if I well interpret the log for example of node with consitentid 7 what I
see is a constant increment of dirty pages:
2019-02-27 17:04:42.206 INFO 41153 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=b3d5b34c-60d3-4c39-af0f-5de6d41686f3, pages=65681,
markPos=FileWALPointer [idx=16, fileOff=24231459, len=75645],
walSegmentsCleared=6, walSegmentsCovered=[10 - 15], markDuration=74ms,
pagesWrite=1429ms, fsync=101ms, total=1604ms]
2019-02-27 17:07:42.244 INFO 41153 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=83cc0c0a-9427-465d-a290-f8a56c366528, pages=69139,
markPos=FileWALPointer [idx=22, fileOff=41402244, len=75645],
walSegmentsCleared=6, walSegmentsCovered=[16 - 21], markDuration=26ms,
pagesWrite=1521ms, fsync=101ms, total=1648ms]
2019-02-27 17:10:42.553 INFO 41153 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=4fe0396f-5ece-4016-a269-119d5ecd8d16, pages=72896,
markPos=FileWALPointer [idx=29, fileOff=6850845, len=75645],
walSegmentsCleared=12, walSegmentsCovered=[22 - 28], markDuration=28ms,
pagesWrite=1713ms, fsync=207ms, total=1948ms]
2019-02-27 17:13:45.214 INFO 41153 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=a309e6a2-865d-439e-ab6d-11ef9641c8b0, pages=75381,
markPos=FileWALPointer [idx=35, fileOff=48279077, len=75645],
walSegmentsCleared=7, walSegmentsCovered=[29 - 34], markDuration=44ms,
pagesWrite=4446ms, fsync=112ms, total=4602ms]
2019-02-27 17:16:42.728 INFO 41153 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=0299f7d3-8ed0-4f0b-8499-14e46d3bef13, pages=78944,
markPos=FileWALPointer [idx=42, fileOff=39209416, len=75645],
walSegmentsCleared=6, walSegmentsCovered=[35 - 41], markDuration=28ms,
pagesWrite=1989ms, fsync=95ms, total=2112ms]
2019-02-27 17:19:40.829 INFO 41153 --- [r-stripe-10-#27]
o.a.i.i.p.c.p.pagemem.PageMemoryImpl : Throttling is applied to page
modifications [percentOfPartTime=0.31, markDirty=22777 pages/sec,
checkpointWrite=11469 pages/sec, estIdealMarkDirty=226519 pages/sec,
curDirty=0.00, maxDirty=0.07, avgParkTime=122057 ns, pages: (total=82206,
evicted=0, written=10035, synced=0, cpBufUsed=27, cpBufTotal=518215)]
2019-02-27 17:19:50.524 INFO 41153 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=f323c2fd-d358-469e-8e12-d2ae1391a546, pages=82206,
markPos=FileWALPointer [idx=49, fileOff=42053202, len=75645],
walSegmentsCleared=8, walSegmentsCovered=[42 - 48], markDuration=27ms,
pagesWrite=9749ms, fsync=123ms, total=9899ms]
2019-02-27 17:22:40.885 INFO 41153 --- [r-stripe-11-#28]
o.a.i.i.p.c.p.pagemem.PageMemoryImpl : Throttling is applied to page
modifications [percentOfPartTime=0.23, markDirty=14449 pages/sec,
checkpointWrite=6729 pages/sec, estIdealMarkDirty=166922 pages/sec,
curDirty=0.00, maxDirty=0.09, avgParkTime=108997 ns, pages: (total=68092,
evicted=0, written=13266, synced=0, cpBufUsed=12, cpBufTotal=518215)]
I try to explain better:
I see at every finished checkpoint a constant increease of dirty pages and a
consequent duration increase of checkpoint:
1) at 17:04:42.206 dirty pages are 65681 with a delay of checkpoint of
1604ms
2) at 17:19:50.524 dirty pages are 82206 with a delay of checkpoint of
9899ms
So in only 15 minutes with constant rate of 4000 entry/sec I observe a
increment of delay of about 9 times respect to start of simulation and the
consequent apllication of throttling how reported in the log at 17:19:40.
I expected a constant number of dirty page and a consequent constant delay
of checkpoint.
Do you have any suggestion about that?
Thanks a lot.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Performance degradation in case of high volumes
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello Antonio!
I have reviewed your logs but don't understand what is the issue here. Can
you please clarify what is the problem once the warning went away?
Of course you should expect eventual slowdown since the height of B-tree
will still increase with time.
Regards,
--
Ilya Kasnacheev
чт, 28 февр. 2019 г. в 15:27, Antonio Conforti <an...@sia.eu>:
> Hello Ilya,
> there was a misunderstanding.
> The warn log message was the explanation how I set the value 66 as max
> payload.
> After setting the environment variable IGNITE_MAX_INDEX_PAYLOAD_SIZE the
> warning message disappeared.
> But the test seems failed.
> You can see my evaluations in the message above.
>
> Thanks a lot and I look forward to hearing from you.
>
> Regards
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Performance degradation in case of high volumes
Posted by Antonio Conforti <an...@sia.eu>.
Hello Ilya,
there was a misunderstanding.
The warn log message was the explanation how I set the value 66 as max
payload.
After setting the environment variable IGNITE_MAX_INDEX_PAYLOAD_SIZE the
warning message disappeared.
But the test seems failed.
You can see my evaluations in the message above.
Thanks a lot and I look forward to hearing from you.
Regards
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Performance degradation in case of high volumes
Posted by "ilya.kasnacheev" <il...@gmail.com>.
Hello!
I'm afraid you will have to increase IGNITE_MAX_INDEX_PAYLOAD_SIZE until you
no longer see "Indexed columns of a row cannot be fully inlined" warning.
Can you try?
Alternatively you can try specifying it with
cacheCfg.setSqlIndexMaxInlineSize().
Regards,
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Performance degradation in case of high volumes
Posted by Antonio Conforti <an...@sia.eu>.
Hello Ilya.
Thank you so much for your answer.
I tried your solution in the same scenario:
- PARTITIONED cache, SQL indexed by key and by a field (type long, the last
key field) descending
- checkpoint frequency restored to default
- 4000 msg/sec rate (fixed)
I set the environment variable IGNITE_MAX_INDEX_PAYLOAD_SIZE to 66:
(WARN 59726 --- [0-#17%Datafeed%] o.a.i.i.p.query.h2.database.H2TreeIndex :
<DF_CMF_QUOTE> Indexed columns of a row cannot be fully inlined into index
what may lead to slowdown due to additional data page reads, increase index
inline size if needed (set system property IGNITE_MAX_INDEX_PAYLOAD_SIZE
with recommended size (be aware it will be used by default for all indexes
without explicit inline size)) [cacheName=IGN_DF_CMF_QUOTE,
tableName=DF_CMF_QUOTE, idxName=_key_PK, idxCols=(_KEY), idxType=PRIMARY
KEY, curSize=10, recommendedInlineSize=66])
in all cluster nodes and I ran the test again.
We had to stop the test after 3 milion messages and what I could notice was
that the checkpoints collected a number of pages that grew over and over.
This increment grew much faster than in previous tests
(IGNITE_MAX_INDEX_PAYLOAD_SIZE not set).
So, I couldn't detect the performance degradation (because I had to stop the
test necessarily) but the increasing number of pages in every checkpoints
let me expect it will happen.
I attach the server logs of this test.
log_ignite_190227_setIndexPayload.gz
<http://apache-ignite-users.70518.x6.nabble.com/file/t2315/log_ignite_190227_setIndexPayload.gz>
What do you think about ?
Thanks a lot
Regards
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Performance degradation in case of high volumes
Posted by Justin Ji <bi...@foxmail.com>.
Thank for your reply, the project was created, you can refer to :
https://github.com/RedBlackTreei/streamer.git
Related post:
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Data-Streamer-Hung-after-a-period-td21161.html
Looking forward to your reply!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Performance degradation in case of high volumes
Posted by Justin Ji <bi...@foxmail.com>.
Thank for your reply, the project was created, you can refer to :
https://github.com/RedBlackTreei/streamer.git
Related post:
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Data-Streamer-Hung-after-a-period-td21161.html
Looking forward to your reply!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Performance degradation in case of high volumes
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
It's hard to say. Maybe the problem is caused by some other index.
Can you make a reproducer project which will exhibit this behavior so that
we could check?
Regards,
--
Ilya Kasnacheev
чт, 28 февр. 2019 г. в 17:41, Justin Ji <bi...@foxmail.com>:
> Ilya -
>
> I have tried to increase QueryIndex.setInlineSize and
> CacheConfiguration.setSqlIndexMaxInlineSize to 128 256 and 512, but the
> performace became worse.
>
> Do I miss some configuraiton?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Performance degradation in case of high volumes
Posted by Justin Ji <bi...@foxmail.com>.
Ilya -
I have tried to increase QueryIndex.setInlineSize and
CacheConfiguration.setSqlIndexMaxInlineSize to 128 256 and 512, but the
performace became worse.
Do I miss some configuraiton?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Performance degradation in case of high volumes
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Since your key is composite, it's hard to estimate its length (BinaryObject
has a lot of overhead). It's recommended to increase
IGNITE_MAX_INDEX_PAYLOAD_SIZE
until you no longer see this problem. Please try e.g. 128.
You can also try specifying that via setSqlIndexMaxInlineSize().
Regards,
--
Ilya Kasnacheev
чт, 28 февр. 2019 г. в 16:21, BinaryTree <bi...@foxmail.com>:
> Hi Ilya -
> First of all, thank for your reply!
> Here is my cache configuration:
>
> private static CacheConfiguration<DpKey, BinaryObject> getCacheConfiguration(IgniteConfiguration cfg) {
>
> CacheConfiguration<DpKey, BinaryObject> cacheCfg = new CacheConfiguration();
> cacheCfg.setName(IgniteCacheKey.DATA_POINT_NEW.getCode());
> cacheCfg.setCacheMode(CacheMode.PARTITIONED);
> cacheCfg.setBackups(1);
> cacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
> cacheCfg.setDataRegionName(Constants.FIVE_GB_PERSISTENCE_REGION);
> cacheCfg.setCacheStoreFactory(FactoryBuilder.factoryOf(DataPointCacheStore.class));
> cacheCfg.setWriteThrough(true);
> cacheCfg.setWriteBehindEnabled(true);
> cacheCfg.setWriteBehindFlushThreadCount(2);
> cacheCfg.setWriteBehindFlushFrequency(15 * 1000);
> cacheCfg.setWriteBehindFlushSize(409600);
> cacheCfg.setWriteBehindBatchSize(1024);
> cacheCfg.setStoreKeepBinary(true);
> cacheCfg.setQueryParallelism(16);
>
> //2M
> cacheCfg.setRebalanceBatchSize(2 * 1024 * 1024);
> cacheCfg.setRebalanceThrottle(100);
>
> cacheCfg.setSqlIndexMaxInlineSize(256);
>
> List<QueryEntity> entities = getQueryEntities();
> cacheCfg.setQueryEntities(entities);
>
> CacheKeyConfiguration cacheKeyConfiguration = new CacheKeyConfiguration(DpKey.class);
> cacheCfg.setKeyConfiguration(cacheKeyConfiguration);
>
> RendezvousAffinityFunction affinityFunction = new RendezvousAffinityFunction();
> affinityFunction.setPartitions(128);
> affinityFunction.setExcludeNeighbors(true);
> cacheCfg.setAffinity(affinityFunction);
>
>
> cfg.setCacheConfiguration(cacheCfg);
> return cacheCfg;
> }
>
>
> private static List<QueryEntity> getQueryEntities() {
> List<QueryEntity> entities = Lists.newArrayList();
>
> //配置可见(可被查询)字段
> QueryEntity entity = new QueryEntity(DpKey.class.getName(), DpCache.class.getName());
> entity.setTableName(IgniteTableKey.T_DATA_POINT_NEW.getCode());
>
> LinkedHashMap<String, String> map = new LinkedHashMap<>();
> map.put("id", "java.lang.String");
> map.put("gmtCreate", "java.lang.Long");
> map.put("gmtModified", "java.lang.Long");
> map.put("devId", "java.lang.String");
> map.put("dpId", "java.lang.Integer");
> map.put("code", "java.lang.String");
> map.put("name", "java.lang.String");
> map.put("customName", "java.lang.String");
> map.put("mode", "java.lang.String");
> map.put("type", "java.lang.String");
> map.put("value", "java.lang.String");
> map.put("rawValue", byte[].class.getName());
> map.put("time", "java.lang.Long");
> map.put("status", "java.lang.Boolean");
> map.put("uuid", "java.lang.String");
>
> entity.setFields(map);
>
> //配置索引信息
> QueryIndex devIdIdx = new QueryIndex("devId");
> devIdIdx.setName("idx_devId");
> devIdIdx.setInlineSize(32);
> List<QueryIndex> indexes = Lists.newArrayList(devIdIdx);
> entity.setIndexes(indexes);
>
> entities.add(entity);
>
> return entities;
> }
>
> public class DpKey implements Serializable {
> private String key;
> @AffinityKeyMapped
> private String devId;
>
> public DpKey() {
> }
>
> public DpKey(String key, String devId) {
> this.key = key;
> this.devId = devId;
> }
>
> public String getKey() {
> return this.key;
> }
>
> public void setKey(String key) {
> this.key = key;
> }
>
> public String getDevId() {
> return this.devId;
> }
>
> public void setDevId(String devId) {
> this.devId = devId;
> }
>
> public boolean equals(Object o) {
> if (this == o) {
> return true;
> } else if (o != null && this.getClass() == o.getClass()) {
> DpKey key = (DpKey)o;
> return this.key.equals(key.key);
> } else {
> return false;
> }
> }
>
> public int hashCode() {
> return this.key.hashCode();
> }
> }
>
> And I have described my issue in this post and some tests I have done :
>
> http://apache-ignite-users.70518.x6.nabble.com/Ignite-Data-Streamer-Hung-after-a-period-td21161.html
>
> ------------------ 原始邮件 ------------------
> *发件人:* "ilya.kasnacheev"<il...@gmail.com>;
> *发送时间:* 2019年2月28日(星期四) 晚上9:03
> *收件人:* "user"<us...@ignite.apache.org>;
> *主题:* Re: Performance degradation in case of high volumes
>
> Hello Justin!
>
> Ignite 2.6 does have IGNITE_MAX_INDEX_PAYLOAD_SIZE system property.
>
> We are talking about primary key here. What is your primary key type? What
> other indexes do you have? Can you provide complete configuration for
> affected tables (including POJOs if applicable?)
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 28 февр. 2019 г. в 15:29, Justin Ji <bi...@foxmail.com>:
>
>> Ilya -
>>
>> I use ignite 2.6.0, does not have IGNITE_MAX_INDEX_PAYLOAD_SIZE system
>> property.
>> But our index field has a fixed length:25 characters, so where can I find
>> the algorithm to calculate the 'index inline size'.
>>
>> Looking forward to your reply.
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
回复: Performance degradation in case of high volumes
Posted by BinaryTree <bi...@foxmail.com>.
Hi Ilya -
First of all, thank for your reply!
Here is my cache configuration:
private static CacheConfiguration<DpKey, BinaryObject> getCacheConfiguration(IgniteConfiguration cfg) {
CacheConfiguration<DpKey, BinaryObject> cacheCfg = new CacheConfiguration();
cacheCfg.setName(IgniteCacheKey.DATA_POINT_NEW.getCode());
cacheCfg.setCacheMode(CacheMode.PARTITIONED);
cacheCfg.setBackups(1);
cacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
cacheCfg.setDataRegionName(Constants.FIVE_GB_PERSISTENCE_REGION);
cacheCfg.setCacheStoreFactory(FactoryBuilder.factoryOf(DataPointCacheStore.class));
cacheCfg.setWriteThrough(true);
cacheCfg.setWriteBehindEnabled(true);
cacheCfg.setWriteBehindFlushThreadCount(2);
cacheCfg.setWriteBehindFlushFrequency(15 * 1000);
cacheCfg.setWriteBehindFlushSize(409600);
cacheCfg.setWriteBehindBatchSize(1024);
cacheCfg.setStoreKeepBinary(true);
cacheCfg.setQueryParallelism(16);
//2M
cacheCfg.setRebalanceBatchSize(2 * 1024 * 1024);
cacheCfg.setRebalanceThrottle(100);
cacheCfg.setSqlIndexMaxInlineSize(256);
List<QueryEntity> entities = getQueryEntities();
cacheCfg.setQueryEntities(entities);
CacheKeyConfiguration cacheKeyConfiguration = new CacheKeyConfiguration(DpKey.class);
cacheCfg.setKeyConfiguration(cacheKeyConfiguration);
RendezvousAffinityFunction affinityFunction = new RendezvousAffinityFunction();
affinityFunction.setPartitions(128);
affinityFunction.setExcludeNeighbors(true);
cacheCfg.setAffinity(affinityFunction);
cfg.setCacheConfiguration(cacheCfg);
return cacheCfg;
}
private static List<QueryEntity> getQueryEntities() {
List<QueryEntity> entities = Lists.newArrayList();
//配置可见(可被查询)字段
QueryEntity entity = new QueryEntity(DpKey.class.getName(), DpCache.class.getName());
entity.setTableName(IgniteTableKey.T_DATA_POINT_NEW.getCode());
LinkedHashMap<String, String> map = new LinkedHashMap<>();
map.put("id", "java.lang.String");
map.put("gmtCreate", "java.lang.Long");
map.put("gmtModified", "java.lang.Long");
map.put("devId", "java.lang.String");
map.put("dpId", "java.lang.Integer");
map.put("code", "java.lang.String");
map.put("name", "java.lang.String");
map.put("customName", "java.lang.String");
map.put("mode", "java.lang.String");
map.put("type", "java.lang.String");
map.put("value", "java.lang.String");
map.put("rawValue", byte[].class.getName());
map.put("time", "java.lang.Long");
map.put("status", "java.lang.Boolean");
map.put("uuid", "java.lang.String");
entity.setFields(map);
//配置索引信息
QueryIndex devIdIdx = new QueryIndex("devId");
devIdIdx.setName("idx_devId");
devIdIdx.setInlineSize(32);
List<QueryIndex> indexes = Lists.newArrayList(devIdIdx);
entity.setIndexes(indexes);
entities.add(entity);
return entities;
}
public class DpKey implements Serializable {
private String key;
@AffinityKeyMapped
private String devId;
public DpKey() {
}
public DpKey(String key, String devId) {
this.key = key;
this.devId = devId;
}
public String getKey() {
return this.key;
}
public void setKey(String key) {
this.key = key;
}
public String getDevId() {
return this.devId;
}
public void setDevId(String devId) {
this.devId = devId;
}
public boolean equals(Object o) {
if (this == o) {
return true;
} else if (o != null && this.getClass() == o.getClass()) {
DpKey key = (DpKey)o;
return this.key.equals(key.key);
} else {
return false;
}
}
public int hashCode() {
return this.key.hashCode();
}
}
And I have described my issue in this post and some tests I have done :
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Data-Streamer-Hung-after-a-period-td21161.html
------------------ 原始邮件 ------------------
发件人: "ilya.kasnacheev"<il...@gmail.com>;
发送时间: 2019年2月28日(星期四) 晚上9:03
收件人: "user"<us...@ignite.apache.org>;
主题: Re: Performance degradation in case of high volumes
Hello Justin!
Ignite 2.6 does have IGNITE_MAX_INDEX_PAYLOAD_SIZE system property.
We are talking about primary key here. What is your primary key type? What other indexes do you have? Can you provide complete configuration for affected tables (including POJOs if applicable?)
Regards,
--
Ilya Kasnacheev
чт, 28 февр. 2019 г. в 15:29, Justin Ji <bi...@foxmail.com>:
Ilya -
I use ignite 2.6.0, does not have IGNITE_MAX_INDEX_PAYLOAD_SIZE system
property.
But our index field has a fixed length:25 characters, so where can I find
the algorithm to calculate the 'index inline size'.
Looking forward to your reply.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Performance degradation in case of high volumes
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello Justin!
Ignite 2.6 does have IGNITE_MAX_INDEX_PAYLOAD_SIZE system property.
We are talking about primary key here. What is your primary key type? What
other indexes do you have? Can you provide complete configuration for
affected tables (including POJOs if applicable?)
Regards,
--
Ilya Kasnacheev
чт, 28 февр. 2019 г. в 15:29, Justin Ji <bi...@foxmail.com>:
> Ilya -
>
> I use ignite 2.6.0, does not have IGNITE_MAX_INDEX_PAYLOAD_SIZE system
> property.
> But our index field has a fixed length:25 characters, so where can I find
> the algorithm to calculate the 'index inline size'.
>
> Looking forward to your reply.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Performance degradation in case of high volumes
Posted by Justin Ji <bi...@foxmail.com>.
Ilya -
I use ignite 2.6.0, does not have IGNITE_MAX_INDEX_PAYLOAD_SIZE system
property.
But our index field has a fixed length:25 characters, so where can I find
the algorithm to calculate the 'index inline size'.
Looking forward to your reply.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Performance degradation in case of high volumes
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
It is possible that your primary key would not fit into index and walking
index tree will put unnecessary strain on persistent page memory.
On 2.7 there will be a warning about that in log.
Please try to tune IGNITE_MAX_INDEX_PAYLOAD_SIZE system property since it's
the only available means of changing that value. Try to put something line
32 or 64 there, see if it helps.
Regards,
--
Ilya Kasnacheev
вт, 26 февр. 2019 г. в 20:24, Antonio Conforti <an...@sia.eu>:
> Hello,
>
>
> I recap the scenario of benchmark:
>
> 1) Constant submission of 4000 entries per second where every entry is an
> add (the key contains a field updatetime and changes for every entry).
> 2) The benchmark starts with no data in cache and the entries are submitted
> from an Ignite client node in the cluster using the StreamVisitor.addData()
> method (located on HOST1).
> 3) The cluster is composed of a total of 8 ignite server nodes: server
> nodes
> with consistence ID 1,3,5,7 on HOST1 and server nodes with consistence ID
> 2,4,6,8 on HOST2;
>
>
> Below the general configuration:
>
> Cache configuration:
>
> 1. TRANSACTIONAL
> 2. partitioned
> 3. with backup 1 (and affinity with exclude neighbours enabled)
> 4. write synchronization mode FULL_ASYNC
> 5. indexed on key and value (and enabled to SQL inquiry)
>
>
> We have also configured:
> 1. failureDetectionTimeout to 120000msec
> 2. Data region (only 1):
> a. Persistence enabled
> b. max size 8 GB
> c. checkpointPageBufferSize 2 GB
> 3. WAL mode LOG_ONLY
> 4. disabled WAL archiving (WAL path and the WAL archive path to
> the same
> value)
> 5. Pages Writes Throttling enabled
>
>
> I run also a second scenario with frequency set to 600000 (10minutes) as
> suggested and with direct IO enabled and WAL mode set to NONE.
>
>
> For problem of space you can find logs and configuration file of only
> server
> node 1 attached for both scenarios as described below:
>
> 1) folder 20190222: log and config for scenario 1)
> 2) folder 20190225: log and config for scenario 2)
> 3) IGN_DF_CMF_QUOTE_PK.java: key entity inserted in cache
> 4) IGN_DF_CMF_QUOTE.java: entity data inserted in cache
>
> case-Ignite.zip
> <http://apache-ignite-users.70518.x6.nabble.com/file/t2315/case-Ignite.zip>
>
>
> If you need another benchmark with specific configuration, just let me
> know.
>
> Thanks.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Performance degradation in case of high volumes
Posted by Antonio Conforti <an...@sia.eu>.
Hello,
I recap the scenario of benchmark:
1) Constant submission of 4000 entries per second where every entry is an
add (the key contains a field updatetime and changes for every entry).
2) The benchmark starts with no data in cache and the entries are submitted
from an Ignite client node in the cluster using the StreamVisitor.addData()
method (located on HOST1).
3) The cluster is composed of a total of 8 ignite server nodes: server nodes
with consistence ID 1,3,5,7 on HOST1 and server nodes with consistence ID
2,4,6,8 on HOST2;
Below the general configuration:
Cache configuration:
1. TRANSACTIONAL
2. partitioned
3. with backup 1 (and affinity with exclude neighbours enabled)
4. write synchronization mode FULL_ASYNC
5. indexed on key and value (and enabled to SQL inquiry)
We have also configured:
1. failureDetectionTimeout to 120000msec
2. Data region (only 1):
a. Persistence enabled
b. max size 8 GB
c. checkpointPageBufferSize 2 GB
3. WAL mode LOG_ONLY
4. disabled WAL archiving (WAL path and the WAL archive path to
the same
value)
5. Pages Writes Throttling enabled
I run also a second scenario with frequency set to 600000 (10minutes) as
suggested and with direct IO enabled and WAL mode set to NONE.
For problem of space you can find logs and configuration file of only server
node 1 attached for both scenarios as described below:
1) folder 20190222: log and config for scenario 1)
2) folder 20190225: log and config for scenario 2)
3) IGN_DF_CMF_QUOTE_PK.java: key entity inserted in cache
4) IGN_DF_CMF_QUOTE.java: entity data inserted in cache
case-Ignite.zip
<http://apache-ignite-users.70518.x6.nabble.com/file/t2315/case-Ignite.zip>
If you need another benchmark with specific configuration, just let me know.
Thanks.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Performance degradation in case of high volumes
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Can you please provide full log? It's hard to say what is going on here.
Can you please also share your Ignite/cache configuration and describe your
work load?
Regards,
--
Ilya Kasnacheev
пн, 25 февр. 2019 г. в 17:32, Dodong Juan <do...@gmail.com>:
> I did observe the same thing with ignite and was not able to resolve it at
> all.
> So I am very interested on a resolution for this too.
>
> > On Feb 25, 2019, at 9:30 AM, Antonio Conforti <an...@sia.eu>
> wrote:
> >
> > Hello,
> >
> > first of all thanks for your advice.
> >
> > Before reading your post I runned another session (at 15:45 about) of
> test
> > with the same rate before changing the frequency of checkpoint you
> suggested
> > but with data already loaded.
> >
> >
> > I observed that performance degrades suddenly and not after a while
> (about
> > 45 minutes) as observed in my first session when the cache was empty.
> > Below you can see the statistics:
> >
> > 2019-02-22 15:49:03.992 INFO 5271 --- [oint-thread-#67]
> > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> > [cpId=ab04c732-a72c-4398-8a09-31f05243a7ad, pages=129721,
> > markPos=FileWALPointer [idx=470, fileOff=35838781, len=79426],
> > walSegmentsCleared=11, walSegmentsCovered=[460 - 469],
> markDuration=387ms,
> > pagesWrite=1149ms, fsync=227229ms, total=228765ms]
> > 2019-02-22 15:53:25.681 INFO 5271 --- [oint-thread-#67]
> > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> > [cpId=cf6e6d68-ac4d-4445-becb-b7bcbb5aee32, pages=178943,
> > markPos=FileWALPointer [idx=484, fileOff=61358243, len=79426],
> > walSegmentsCleared=14, walSegmentsCovered=[470 - 483],
> markDuration=1316ms,
> > pagesWrite=1043ms, fsync=259329ms, total=261688ms]
> > 2019-02-22 15:56:07.878 INFO 5271 --- [oint-thread-#67]
> > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> > [cpId=86154f6f-907e-4972-b0f1-f86be06c3a40, pages=44514,
> > markPos=FileWALPointer [idx=487, fileOff=66267301, len=79426],
> > walSegmentsCleared=0, walSegmentsCovered=[484 - 486], markDuration=888ms,
> > pagesWrite=239ms, fsync=161070ms, total=162197ms]
> >
> > What looks strange in this second session is that I see a delay between
> 228
> > and 260 sec to write 129721 / 178943 pages.
> > I don't observe this behaviour when I start from scratch (with no data in
> > cache): it usually takes about 45 minutes before observing this
> performance
> > degradation.
> > Any clues on what can be the cause?
> >
> >
> > I started today another test session with the suggested frequency
> > (checkpointingFrequency=600000) and I also configured direct IO and set
> the
> > WAL mode to NONE as suggested in tuning performance but the result
> observed
> > is quite the same
> >
> >
> >
> > 2019-02-25 09:34:02.284 INFO 1208 --- [oint-thread-#66]
> > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> > [cpId=a9423ba9-9d93-40a3-bedb-7379a617f716, pages=78,
> markPos=FileWALPointer
> > [idx=0, fileOff=0, len=0], walSegmentsCleared=0, walSegmentsCovered=[],
> > markDuration=13ms, pagesWrite=15ms, fsync=11ms, total=39ms]
> > 2019-02-25 10:04:04.082 INFO 1208 --- [oint-thread-#66]
> > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> > [cpId=bd76a704-e125-42e1-b2a8-d285cf15246d, pages=52252,
> > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> > walSegmentsCovered=[], markDuration=144ms, pagesWrite=1560ms,
> fsync=123ms,
> > total=1827ms]
> > 2019-02-25 10:14:27.275 INFO 1208 --- [oint-thread-#66]
> > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> > [cpId=a10f5483-050c-4678-98bd-deca7695e782, pages=368810,
> > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> > walSegmentsCovered=[], markDuration=323ms, pagesWrite=24669ms,
> fsync=21ms,
> > total=25013ms]
> > 2019-02-25 10:24:55.492 INFO 1208 --- [oint-thread-#66]
> > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> > [cpId=bfab59c6-a1a0-4391-92e5-a23b6c416625, pages=382555,
> > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> > walSegmentsCovered=[], markDuration=225ms, pagesWrite=52912ms,
> fsync=93ms,
> > total=53230ms]
> > 2019-02-25 10:35:22.512 INFO 1208 --- [oint-thread-#66]
> > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> > [cpId=feff29e3-db80-4002-82f8-2011e4eca12e, pages=436129,
> > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> > walSegmentsCovered=[], markDuration=70ms, pagesWrite=80164ms, fsync=7ms,
> > total=80241ms]
> > 2019-02-25 10:45:34.524 INFO 1208 --- [oint-thread-#66]
> > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> > [cpId=7e3f181b-21d8-4dd1-a018-09711142caf4, pages=384415,
> > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> > walSegmentsCovered=[], markDuration=56ms, pagesWrite=92186ms, fsync=3ms,
> > total=92245ms]
> > 2019-02-25 10:55:43.139 INFO 1208 --- [oint-thread-#66]
> > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> > [cpId=a77d48d8-f806-4a48-bfbf-d1afac7d16e3, pages=161643,
> > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> > walSegmentsCovered=[], markDuration=73ms, pagesWrite=100765ms,
> fsync=16ms,
> > total=100854ms]
> > 2019-02-25 11:06:04.437 INFO 1208 --- [oint-thread-#66]
> > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> > [cpId=dbf6a8ab-1c47-4cdd-96f4-fec852dc4fe0, pages=129477,
> > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> > walSegmentsCovered=[], markDuration=60ms, pagesWrite=121933ms,
> fsync=154ms,
> > total=122147ms]
> > 2019-02-25 11:16:08.333 INFO 1208 --- [oint-thread-#66]
> > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> > [cpId=74924992-3dc4-42db-971c-b9030f41d0c5, pages=97511,
> > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> > walSegmentsCovered=[], markDuration=1676ms, pagesWrite=124336ms,
> fsync=25ms,
> > total=126037ms]
> > 2019-02-25 11:18:53.622 INFO 1208 --- [oint-thread-#66]
> > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> > [cpId=7c60efc2-4007-4be2-b3c8-3912d2cbd651, pages=19277,
> > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> > walSegmentsCovered=[], markDuration=12ms, pagesWrite=47883ms,
> fsync=344ms,
> > total=48239ms]
> >
> >
> > Do you have any other suggestions considering also the behaviour
> described
> > above?
> >
> > Thanks,
> > Antonio
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>
Re: Performance degradation in case of high volumes
Posted by Dodong Juan <do...@gmail.com>.
I did observe the same thing with ignite and was not able to resolve it at all.
So I am very interested on a resolution for this too.
> On Feb 25, 2019, at 9:30 AM, Antonio Conforti <an...@sia.eu> wrote:
>
> Hello,
>
> first of all thanks for your advice.
>
> Before reading your post I runned another session (at 15:45 about) of test
> with the same rate before changing the frequency of checkpoint you suggested
> but with data already loaded.
>
>
> I observed that performance degrades suddenly and not after a while (about
> 45 minutes) as observed in my first session when the cache was empty.
> Below you can see the statistics:
>
> 2019-02-22 15:49:03.992 INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=ab04c732-a72c-4398-8a09-31f05243a7ad, pages=129721,
> markPos=FileWALPointer [idx=470, fileOff=35838781, len=79426],
> walSegmentsCleared=11, walSegmentsCovered=[460 - 469], markDuration=387ms,
> pagesWrite=1149ms, fsync=227229ms, total=228765ms]
> 2019-02-22 15:53:25.681 INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=cf6e6d68-ac4d-4445-becb-b7bcbb5aee32, pages=178943,
> markPos=FileWALPointer [idx=484, fileOff=61358243, len=79426],
> walSegmentsCleared=14, walSegmentsCovered=[470 - 483], markDuration=1316ms,
> pagesWrite=1043ms, fsync=259329ms, total=261688ms]
> 2019-02-22 15:56:07.878 INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=86154f6f-907e-4972-b0f1-f86be06c3a40, pages=44514,
> markPos=FileWALPointer [idx=487, fileOff=66267301, len=79426],
> walSegmentsCleared=0, walSegmentsCovered=[484 - 486], markDuration=888ms,
> pagesWrite=239ms, fsync=161070ms, total=162197ms]
>
> What looks strange in this second session is that I see a delay between 228
> and 260 sec to write 129721 / 178943 pages.
> I don't observe this behaviour when I start from scratch (with no data in
> cache): it usually takes about 45 minutes before observing this performance
> degradation.
> Any clues on what can be the cause?
>
>
> I started today another test session with the suggested frequency
> (checkpointingFrequency=600000) and I also configured direct IO and set the
> WAL mode to NONE as suggested in tuning performance but the result observed
> is quite the same
>
>
>
> 2019-02-25 09:34:02.284 INFO 1208 --- [oint-thread-#66]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=a9423ba9-9d93-40a3-bedb-7379a617f716, pages=78, markPos=FileWALPointer
> [idx=0, fileOff=0, len=0], walSegmentsCleared=0, walSegmentsCovered=[],
> markDuration=13ms, pagesWrite=15ms, fsync=11ms, total=39ms]
> 2019-02-25 10:04:04.082 INFO 1208 --- [oint-thread-#66]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=bd76a704-e125-42e1-b2a8-d285cf15246d, pages=52252,
> markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> walSegmentsCovered=[], markDuration=144ms, pagesWrite=1560ms, fsync=123ms,
> total=1827ms]
> 2019-02-25 10:14:27.275 INFO 1208 --- [oint-thread-#66]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=a10f5483-050c-4678-98bd-deca7695e782, pages=368810,
> markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> walSegmentsCovered=[], markDuration=323ms, pagesWrite=24669ms, fsync=21ms,
> total=25013ms]
> 2019-02-25 10:24:55.492 INFO 1208 --- [oint-thread-#66]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=bfab59c6-a1a0-4391-92e5-a23b6c416625, pages=382555,
> markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> walSegmentsCovered=[], markDuration=225ms, pagesWrite=52912ms, fsync=93ms,
> total=53230ms]
> 2019-02-25 10:35:22.512 INFO 1208 --- [oint-thread-#66]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=feff29e3-db80-4002-82f8-2011e4eca12e, pages=436129,
> markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> walSegmentsCovered=[], markDuration=70ms, pagesWrite=80164ms, fsync=7ms,
> total=80241ms]
> 2019-02-25 10:45:34.524 INFO 1208 --- [oint-thread-#66]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=7e3f181b-21d8-4dd1-a018-09711142caf4, pages=384415,
> markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> walSegmentsCovered=[], markDuration=56ms, pagesWrite=92186ms, fsync=3ms,
> total=92245ms]
> 2019-02-25 10:55:43.139 INFO 1208 --- [oint-thread-#66]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=a77d48d8-f806-4a48-bfbf-d1afac7d16e3, pages=161643,
> markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> walSegmentsCovered=[], markDuration=73ms, pagesWrite=100765ms, fsync=16ms,
> total=100854ms]
> 2019-02-25 11:06:04.437 INFO 1208 --- [oint-thread-#66]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=dbf6a8ab-1c47-4cdd-96f4-fec852dc4fe0, pages=129477,
> markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> walSegmentsCovered=[], markDuration=60ms, pagesWrite=121933ms, fsync=154ms,
> total=122147ms]
> 2019-02-25 11:16:08.333 INFO 1208 --- [oint-thread-#66]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=74924992-3dc4-42db-971c-b9030f41d0c5, pages=97511,
> markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> walSegmentsCovered=[], markDuration=1676ms, pagesWrite=124336ms, fsync=25ms,
> total=126037ms]
> 2019-02-25 11:18:53.622 INFO 1208 --- [oint-thread-#66]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=7c60efc2-4007-4be2-b3c8-3912d2cbd651, pages=19277,
> markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
> walSegmentsCovered=[], markDuration=12ms, pagesWrite=47883ms, fsync=344ms,
> total=48239ms]
>
>
> Do you have any other suggestions considering also the behaviour described
> above?
>
> Thanks,
> Antonio
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Performance degradation in case of high volumes
Posted by Antonio Conforti <an...@sia.eu>.
Hello,
first of all thanks for your advice.
Before reading your post I runned another session (at 15:45 about) of test
with the same rate before changing the frequency of checkpoint you suggested
but with data already loaded.
I observed that performance degrades suddenly and not after a while (about
45 minutes) as observed in my first session when the cache was empty.
Below you can see the statistics:
2019-02-22 15:49:03.992 INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=ab04c732-a72c-4398-8a09-31f05243a7ad, pages=129721,
markPos=FileWALPointer [idx=470, fileOff=35838781, len=79426],
walSegmentsCleared=11, walSegmentsCovered=[460 - 469], markDuration=387ms,
pagesWrite=1149ms, fsync=227229ms, total=228765ms]
2019-02-22 15:53:25.681 INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=cf6e6d68-ac4d-4445-becb-b7bcbb5aee32, pages=178943,
markPos=FileWALPointer [idx=484, fileOff=61358243, len=79426],
walSegmentsCleared=14, walSegmentsCovered=[470 - 483], markDuration=1316ms,
pagesWrite=1043ms, fsync=259329ms, total=261688ms]
2019-02-22 15:56:07.878 INFO 5271 --- [oint-thread-#67]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=86154f6f-907e-4972-b0f1-f86be06c3a40, pages=44514,
markPos=FileWALPointer [idx=487, fileOff=66267301, len=79426],
walSegmentsCleared=0, walSegmentsCovered=[484 - 486], markDuration=888ms,
pagesWrite=239ms, fsync=161070ms, total=162197ms]
What looks strange in this second session is that I see a delay between 228
and 260 sec to write 129721 / 178943 pages.
I don't observe this behaviour when I start from scratch (with no data in
cache): it usually takes about 45 minutes before observing this performance
degradation.
Any clues on what can be the cause?
I started today another test session with the suggested frequency
(checkpointingFrequency=600000) and I also configured direct IO and set the
WAL mode to NONE as suggested in tuning performance but the result observed
is quite the same
2019-02-25 09:34:02.284 INFO 1208 --- [oint-thread-#66]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=a9423ba9-9d93-40a3-bedb-7379a617f716, pages=78, markPos=FileWALPointer
[idx=0, fileOff=0, len=0], walSegmentsCleared=0, walSegmentsCovered=[],
markDuration=13ms, pagesWrite=15ms, fsync=11ms, total=39ms]
2019-02-25 10:04:04.082 INFO 1208 --- [oint-thread-#66]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=bd76a704-e125-42e1-b2a8-d285cf15246d, pages=52252,
markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
walSegmentsCovered=[], markDuration=144ms, pagesWrite=1560ms, fsync=123ms,
total=1827ms]
2019-02-25 10:14:27.275 INFO 1208 --- [oint-thread-#66]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=a10f5483-050c-4678-98bd-deca7695e782, pages=368810,
markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
walSegmentsCovered=[], markDuration=323ms, pagesWrite=24669ms, fsync=21ms,
total=25013ms]
2019-02-25 10:24:55.492 INFO 1208 --- [oint-thread-#66]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=bfab59c6-a1a0-4391-92e5-a23b6c416625, pages=382555,
markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
walSegmentsCovered=[], markDuration=225ms, pagesWrite=52912ms, fsync=93ms,
total=53230ms]
2019-02-25 10:35:22.512 INFO 1208 --- [oint-thread-#66]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=feff29e3-db80-4002-82f8-2011e4eca12e, pages=436129,
markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
walSegmentsCovered=[], markDuration=70ms, pagesWrite=80164ms, fsync=7ms,
total=80241ms]
2019-02-25 10:45:34.524 INFO 1208 --- [oint-thread-#66]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=7e3f181b-21d8-4dd1-a018-09711142caf4, pages=384415,
markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
walSegmentsCovered=[], markDuration=56ms, pagesWrite=92186ms, fsync=3ms,
total=92245ms]
2019-02-25 10:55:43.139 INFO 1208 --- [oint-thread-#66]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=a77d48d8-f806-4a48-bfbf-d1afac7d16e3, pages=161643,
markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
walSegmentsCovered=[], markDuration=73ms, pagesWrite=100765ms, fsync=16ms,
total=100854ms]
2019-02-25 11:06:04.437 INFO 1208 --- [oint-thread-#66]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=dbf6a8ab-1c47-4cdd-96f4-fec852dc4fe0, pages=129477,
markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
walSegmentsCovered=[], markDuration=60ms, pagesWrite=121933ms, fsync=154ms,
total=122147ms]
2019-02-25 11:16:08.333 INFO 1208 --- [oint-thread-#66]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=74924992-3dc4-42db-971c-b9030f41d0c5, pages=97511,
markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
walSegmentsCovered=[], markDuration=1676ms, pagesWrite=124336ms, fsync=25ms,
total=126037ms]
2019-02-25 11:18:53.622 INFO 1208 --- [oint-thread-#66]
i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
[cpId=7c60efc2-4007-4be2-b3c8-3912d2cbd651, pages=19277,
markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0,
walSegmentsCovered=[], markDuration=12ms, pagesWrite=47883ms, fsync=344ms,
total=48239ms]
Do you have any other suggestions considering also the behaviour described
above?
Thanks,
Antonio
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Performance degradation in case of high volumes
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
It seems that, as time goes, I/O can't catch up with you.
The recommendation here is, probably, to increase checkpoint frequency
value (measured in ms; to do checkpoints less often)
Let's say, set it to 600000 (10 minutes).
The downside here is that in case of crash, node will take more time to
come online.
Regards,
--
Ilya Kasnacheev
пт, 22 февр. 2019 г. в 17:37, Antonio Conforti <an...@sia.eu>:
> Hi support,
>
> I'm running a performance test writing 4000 entry per second on a cache:
> 1. TRANSACTIONAL
> 2. partitioned
> 3. with backup 1 (and affinity with exclude neighbors enabled)
> 4. write synchronization mode FULL_ASYNC
> 5. indexed on key and value (and enabled to SQL inquiry)
>
> Writes are performed by a client node using a data stream with
> StreamVisitor
> and set autoFlushFrequency 1 sec.
>
> We have configured:
> 1. failureDetectionTimeout to 120000msec
> 2. Data region (only 1):
> a. Persistence enabled
> b. max size 8 GB
> c. checkpointPageBufferSize 2 GB
> 3. WAL mode LOG_ONLY
> 4. disabled WAL archiving (WAL path and the WAL archive path to the
> same
> value)
> 5. Pages Writes Throttling enabled
>
>
> After some hour submitting about 20 million entries without problems, the
> client node starts to accuse delays: the queue from the client node Ignite
> reads messages start to grow.
>
> Verifying the logs of server and client node there isn’t any error message
> but from the statistics of WAL high FSYNC values are observed.
>
> Could you help me to understand why inspite a constant rate and a constant
> consumption of cpu of about 30% only after a certain amount of entry it
> seems the server slow down in term of performance?
>
> May be there is some param to tune that I missed?
>
> Below the configuration used for the simulation:
>
> Total server nodes 8 so distributed:
> HOST1 with 4 nodes server and 1 client node on HDD disk
> HOST2 with 4 nodes on HDD disk
>
>
> Both hosts are machines with 16 cores of 256 GB of memory and HHD disk.
>
> The DataStorageConfiguration for each server node is as follows:
>
>
> <property name="dataStorageConfiguration">
> <bean
>
> class="org.apache.ignite.configuration.DataStorageConfiguration">
>
> <property name="writeThrottlingEnabled"
> value="true"
> />
> <property name="defaultDataRegionConfiguration">
> <bean
>
> class="org.apache.ignite.configuration.DataRegionConfiguration">
> <property name="persistenceEnabled"
> value="true" />
> <property name="maxSize"
> value="#{8L
> * 1024 * 1024 * 1024}"/>
> <property
> name="checkpointPageBufferSize"
> value="#{2048L * 1024 *
> 1024}" />
> </bean>
> </property>
>
>
>
> <property name="walMode" value="LOG_ONLY" />
> <property name="walPath" value="wal/path" />
> <property name="walArchivePath" value="wal/path" />
> </bean>
> </property>
>
>
> JVM option used for start each server node:
>
> -server -Xms4g -Xmx8g -XX:+AlwaysPreTouch -XX:+UseG1GC
> -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
>
>
>
> I report the WAL statistics from log of node 1 :
>
> At Simulation start:
> 2019-02-22 10:19:44.195 INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=c115ead9-643d-45e5-be41-cd7ae5caac14, pages=11891,
> markPos=FileWALPointer [idx=1, fileOff=36517886, len=79426],
> walSegmentsCleared=0, walSegmentsCovered=[0], markDuration=34ms,
> pagesWrite=87ms, fsync=1931ms, total=2052ms]
> 2019-02-22 10:22:44.742 INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=d40c9096-2e10-46ca-ae8e-2a39e242b768, pages=66732,
> markPos=FileWALPointer [idx=7, fileOff=63806638, len=79426],
> walSegmentsCleared=7, walSegmentsCovered=[1 - 6], markDuration=98ms,
> pagesWrite=407ms, fsync=2085ms, total=2590ms]
> 2019-02-22 10:25:44.900 INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=5124f12f-6ed8-4ed3-9644-3c58957600ed, pages=70253,
> markPos=FileWALPointer [idx=14, fileOff=47159207, len=79426],
> walSegmentsCleared=6, walSegmentsCovered=[7 - 13], markDuration=98ms,
> pagesWrite=402ms, fsync=2241ms, total=2741ms]
> 2019-02-22 10:28:47.866 INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=9094d36c-cd79-4d30-9b90-e48de78fa3e6, pages=72524,
> markPos=FileWALPointer [idx=21, fileOff=39728290, len=79426],
> walSegmentsCleared=8, walSegmentsCovered=[14 - 20], markDuration=83ms,
> pagesWrite=365ms, fsync=5255ms, total=5703ms]
> 2019-02-22 10:31:53.635 INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=7132d53a-e2a6-4ac8-b1a8-2621cc39c82b, pages=77471,
> markPos=FileWALPointer [idx=28, fileOff=64681287, len=79426],
> walSegmentsCleared=7, walSegmentsCovered=[21 - 27], markDuration=494ms,
> pagesWrite=748ms, fsync=10136ms, total=11472ms]
>
>
> At end of simulation
>
> 2019-02-22 11:52:36.339 INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=dc8369b9-b100-4dd0-bfaf-9a5b13620072, pages=129942,
> markPos=FileWALPointer [idx=309, fileOff=19048810, len=79426],
> walSegmentsCleared=11, walSegmentsCovered=[298 - 308], markDuration=77ms,
> pagesWrite=797ms, fsync=171049ms, total=171923ms]
> 2019-02-22 11:56:24.001 INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=1fb1ced8-a257-4c6f-be97-a1867ba6692e, pages=133096,
> markPos=FileWALPointer [idx=320, fileOff=13420410, len=79426],
> walSegmentsCleared=11, walSegmentsCovered=[309 - 319], markDuration=1707ms,
> pagesWrite=1537ms, fsync=216332ms, total=219576ms]
> 2019-02-22 12:00:23.052 INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=0d48f6d5-7601-4839-b422-55b228978da5, pages=150587,
> markPos=FileWALPointer [idx=332, fileOff=47800048, len=79426],
> walSegmentsCleared=12, walSegmentsCovered=[320 - 331], markDuration=2275ms,
> pagesWrite=752ms, fsync=236023ms, total=239051ms]
> 2019-02-22 12:04:05.562 INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=b23526c4-5121-48a9-9902-522a5ffb3a28, pages=155805,
> markPos=FileWALPointer [idx=345, fileOff=40020477, len=79426],
> walSegmentsCleared=13, walSegmentsCovered=[332 - 344], markDuration=525ms,
> pagesWrite=1324ms, fsync=220654ms, total=222504ms]
> 2019-02-22 12:07:54.005 INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=1bb2a0c0-7a89-47f2-af9f-90e90d44c14b, pages=149055,
> markPos=FileWALPointer [idx=357, fileOff=51666923, len=79426],
> walSegmentsCleared=12, walSegmentsCovered=[345 - 356], markDuration=995ms,
> pagesWrite=1559ms, fsync=225888ms, total=228442ms]
> 2019-02-22 12:11:49.962 INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=19f14a04-842f-409e-9c6b-afb59193e419, pages=153022,
> markPos=FileWALPointer [idx=370, fileOff=16234647, len=79426],
> walSegmentsCleared=13, walSegmentsCovered=[357 - 369], markDuration=1773ms,
> pagesWrite=1044ms, fsync=233139ms, total=235957ms]
> 2019-02-22 12:15:59.332 INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=c1316fb7-1ecc-4358-bf90-a87772969c03, pages=159668,
> markPos=FileWALPointer [idx=383, fileOff=21979375, len=79426],
> walSegmentsCleared=13, walSegmentsCovered=[370 - 382], markDuration=1249ms,
> pagesWrite=1693ms, fsync=246428ms, total=249370ms]
> 2019-02-22 12:20:05.814 INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=a9e394f2-0011-4f89-8b5a-a6ed77774103, pages=156891,
> markPos=FileWALPointer [idx=396, fileOff=3956799, len=79426],
> walSegmentsCleared=13, walSegmentsCovered=[383 - 395], markDuration=1030ms,
> pagesWrite=1275ms, fsync=244176ms, total=246482ms]
> 2019-02-22 12:24:40.217 INFO 5271 --- [oint-thread-#67]
> i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished
> [cpId=0bc71cc5-97c4-4f7d-95c2-430f379eeeb0, pages=148039,
> markPos=FileWALPointer [idx=407, fileOff=57767331, len=79426],
> walSegmentsCleared=11, walSegmentsCovered=[396 - 406], markDuration=323ms,
> pagesWrite=1620ms, fsync=272460ms, total=274403ms]
>
>
> Thanks in advance.
>
> Antonio
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>