You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by dkarachentsev <dk...@gridgain.com> on 2017/08/17 16:13:22 UTC
Re: Performance of persistent store too low when bulb-loading
Hi,
Please take about 5 thread dumps with interval in 1 sec. This will help to
understand what threads busy with.
Thanks!
-Dmitry.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Performance-of-persistent-store-too-low-when-bulb-loading-tp16247p16257.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Performance of persistent store too low when bulb-loading
Posted by Ray <ra...@cisco.com>.
I also had the same problem here.
I'm using ignite-spark's saveParis method(which is also a
IgniteDataStreamer) to ingest 550 million entries of data into Ignite, and
the ingesting speed got slow down after first few minutes and seems stuck
for now with persistent store enabled.
My setup is 4 nodes with 16GB heap size and 32GB off heap size.
Here's the dstat result for about a minute when the ingesting is stuck.
[7l----total-cpu-usage---- ------memory-usage----- -dsk/total- ---paging--
----swap--- --filesystem-
usr sys idl wai hiq siq| used buff cach free| read writ| in out |
used free|files inodes
1 0 99 0 0 0|89.9G 3970M 157G 507G| 30k 967k| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 12M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 9548k| 0 0 | 0
0 | 8176 453k
0 0 96 3 0 0|89.9G 3970M 157G 507G| 0 18M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 17M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 17M| 0 0 | 0
0 | 8176 453k
0 0 96 3 0 0|89.9G 3970M 157G 507G| 0 16M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 15M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 17M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 16M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 18M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 17M| 0 0 | 0
0 | 8176 453k
2 1 94 3 0 0|89.9G 3970M 157G 507G| 0 15M| 0 0 | 0
0 | 8176 453k
1 0 96 3 0 0|89.9G 3970M 157G 507G| 0 24M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 25M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 20M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 26M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 16M| 0 0 | 0
0 | 8176 453k
0 0 96 3 0 0|89.9G 3970M 157G 507G| 0 10M| 0 0 | 0
0 | 8176 453k
1 0 96 3 0 0|89.9G 3970M 157G 507G| 0 24M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 20M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 25M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 31M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 23M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 21M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 21M| 0 0 | 0
0 | 8176 453k
0 0 96 3 0 0|89.9G 3970M 157G 507G| 0 27M| 0 0 | 0
0 | 8176 453k
0 0 96 3 0 0|89.9G 3970M 157G 507G| 0 24M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 27M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 16M| 0 0 | 0
0 | 8176 453k
0 0 96 3 0 0|89.9G 3970M 157G 507G| 0 20M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 28M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 22M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 24M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 27M| 0 0 | 0
0 | 8176 453k
1 0 96 3 0 0|89.9G 3970M 157G 507G| 0 27M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 24M| 0 0 | 0
0 | 8176 453k
1 0 96 3 0 0|89.9G 3970M 157G 507G| 0 29M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 27M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 23M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 17M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 62M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 16M| 0 0 | 0
0 | 8176 453k
0 0 96 3 0 0|89.9G 3970M 157G 507G| 0 17M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 20M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 42M| 0 0 | 0
0 | 8064 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 41M| 0 0 | 0
0 | 8064 453k
0 0 97 2 0 0|89.9G 3970M 157G 507G| 0 28M| 0 0 | 0
0 | 8064 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 36M| 0 0 | 0
0 | 8064 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 31M| 0 0 | 0
0 | 8064 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 22M| 0 0 | 0
0 | 8064 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 20M| 0 0 | 0
0 | 8064 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 28M| 0 0 | 0
0 | 8064 453k
0 0 96 3 0 0|89.9G 3970M 157G 507G| 0 40M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 34M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 18M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 16M| 0 0 | 0
0 | 8176 453k
0 0 96 3 0 0|89.9G 3970M 157G 507G| 0 16M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 22M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 18M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 17M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 24M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 25M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 33M| 0 0 | 0
0 | 8176 453k
1 0 96 3 0 0|89.9G 3970M 157G 507G| 0 53M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 30M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 20M| 0 0 | 0
0 | 8176 453k
0 0 96 3 0 0|89.9G 3970M 157G 507G| 0 24M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 34M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 22M| 0 0 | 0
0 | 8176 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 20M| 0 0 | 0
0 | 8176 453k
0 0 96 3 0 0|89.9G 3970M 157G 507G| 0 17M| 0 0 | 0
0 | 8288 453k
2 1 94 3 0 0|89.9G 3970M 157G 507G| 0 18M| 0 0 | 0
0 | 8288 453k
1 0 96 3 0 0|89.9G 3970M 157G 507G| 0 41M| 0 0 | 0
0 | 8288 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 19M| 0 0 | 0
0 | 8288 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 32M| 0 0 | 0
0 | 8288 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 49M| 0 0 | 0
0 | 8288 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 24M| 0 0 | 0
0 | 8288 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 19M| 0 0 | 0
0 | 8288 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 25M| 0 0 | 0
0 | 8288 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 24M| 0 0 | 0
0 | 8288 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 27M| 0 0 | 0
0 | 8288 453k
0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 21M| 0 0 | 0
0 | 8288 453k
And the GC logs, please scroll down to see today's log.
ignitegc.current
<http://apache-ignite-users.70518.x6.nabble.com/file/t1346/ignitegc.current>
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Performance of persistent store too low when bulb-loading
Posted by dkarachentsev <dk...@gridgain.com>.
Hi,
Had you any chance to get GC and dstat logs?
Thanks!
-Dmitry.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Performance-of-persistent-store-too-low-when-bulb-loading-tp16247p16417.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Performance of persistent store too low when bulb-loading
Posted by dkarachentsev <dk...@gridgain.com>.
Use this command for dstat:
dstat -cmdgs --fs > dstat.txt
Thanks!
-Dmitry.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Performance-of-persistent-store-too-low-when-bulb-loading-tp16247p16345.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Performance of persistent store too low when bulb-loading
Posted by dkarachentsev <dk...@gridgain.com>.
Hi,
Could you please take dstat, Ignite and GC logs and few thread dumps when
application is experiencing slowdown?
Thanks!
-Dmitry.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Performance-of-persistent-store-too-low-when-bulb-loading-tp16247p16342.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Performance of persistent store too low when bulb-loading
Posted by dkarachentsev <dk...@gridgain.com>.
Hi,
The reason of slowdown might be the fact, that Ignite by default consumes
80% of available memory, which forces OS to start a lot of swapping, so all
memory operations limited by disk speed. Try to reduce memory size [1].
[1]
https://apacheignite.readme.io/v2.1/docs/memory-configuration#memory-policies
Thanks!
-Dmitry.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Performance-of-persistent-store-too-low-when-bulb-loading-tp16247p16335.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.