You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by oskoblya <os...@at-consulting.ru> on 2016/05/05 16:17:28 UTC
Slow data loading and high(very) memory usage issues
Hi, I’m trying to load tSV file size ~six gigabytes (contains ~90 million
records) into single Apache Ignite node with allocated heap size = 40
gigabytes. For loading used IgniteDataStreamer. Code and sample (10
records) of data is here:
https://github.com/olegskoblya/traffic-loader.git
Physical machine has 64 GB of RAM, 8 CPU cores (each has 2,25 GHz). Cache
configured for storing data in unlimited off-heap memory
(trafficCacheConf.setMemoryMode(CacheMemoryMode.OFFHEAP_TIERED);
trafficCacheConf.setOffHeapMaxMemory(0);)
After ~18 min physical memory usage increased to 90% and speed of loading
catastrophically slow down (see picture below – GGVisor used in trial/demo
mode with Apache Ignite nodes deployed on single host :) ):
<http://apache-ignite-users.70518.x6.nabble.com/file/n4798/physical_memory_loading_near_end.png>
<http://apache-ignite-users.70518.x6.nabble.com/file/n4798/loading_process.png>
After manually calling GC – nothing changed (in loading speed and memory
usage). Finally file has been loaded but last ~2 million records being
processed the same time as ~88 previous (of course I can’t run sql query on
loaded data because indexes was turned off for better loading performance).
On tab SQL Viewer in column named “Total Memory” placed value 24,7GB for
used Off Heap memory (I mean it is my ~6GB of data “eat” so many?).
<http://apache-ignite-users.70518.x6.nabble.com/file/n4798/off_heap_used.png>
So, it is actually normall that ~6GB of data increases in size four times
after loading into the cache? How being used rest of memory (as 24,7 it is
less then half of 90% used physical memory)? Is it possible to reduce such
memory consuming? And loading 6GB of data to in-memory cache on same machine
in 34 minutes seen as unsatisfactorily.
I will be grateful for any suggestions.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Slow-data-loading-and-high-very-memory-usage-issues-tp4798.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Slow data loading and high(very) memory usage issues
Posted by Denis Magda <dm...@gridgain.com>.
Kevin,
Please follow my suggestions and links given in this thread. Since Java heap size is quite big you should move to off-heap mode and setup garbage collectors properly.
—
Denis
> On May 6, 2016, at 12:12 PM, kevin.zheng <zh...@huawei.com> wrote:
>
> Hi Dsetrakyan,
> I have to 2 servers, with 48g and 32g memory respectively.
> --------------------------------------------------------
> | Heap memory used/max | 29gb/43gb
> --------------------------------------------------------
> Heap memory used/max | 28gb/30gb
>
> cheers,
> Kevin
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Slow-data-loading-and-high-very-memory-usage-issues-tp4798p4816.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Slow data loading and high(very) memory usage issues
Posted by "kevin.zheng" <zh...@huawei.com>.
Hi Dsetrakyan,
I have to 2 servers, with 48g and 32g memory respectively.
--------------------------------------------------------
| Heap memory used/max | 29gb/43gb
--------------------------------------------------------
Heap memory used/max | 28gb/30gb
cheers,
Kevin
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Slow-data-loading-and-high-very-memory-usage-issues-tp4798p4816.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Slow data loading and high(very) memory usage issues
Posted by Dmitriy Setrakyan <ds...@apache.org>.
On Thu, May 5, 2016 at 8:29 PM, kevin.zheng <zh...@huawei.com>
wrote:
> facing similar problem : memory consumption problem.
> loading 47Millions Records, with data streamer method, took 30mins.
> memory usage : 51g.
>
>
What is the total memory you have? Also, can you check in the profiler how
much free memory you have left?
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Slow-data-loading-and-high-very-memory-usage-issues-tp4798p4803.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Slow data loading and high(very) memory usage issues
Posted by "kevin.zheng" <zh...@huawei.com>.
facing similar problem : memory consumption problem.
loading 47Millions Records, with data streamer method, took 30mins.
memory usage : 51g.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Slow-data-loading-and-high-very-memory-usage-issues-tp4798p4803.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Slow data loading and high(very) memory usage issues
Posted by Denis Magda <dm...@gridgain.com>.
You’re welcome.
Is initial data compressed somehow?
In general there is a capacity planning page [1] that describes how calculate approximate size that will be consumed by the grid.
I would suggest measure approximately the size of an object that is stored in the cache and apply this knowledge in the calculation like it’s done in [1].
Let me know if you need more assistance on this.
[1] https://apacheignite.readme.io/docs/capacity-planning <https://apacheignite.readme.io/docs/capacity-planning>
—
Denis
> On May 6, 2016, at 12:43 PM, oskoblya <os...@at-consulting.ru> wrote:
>
> Thank's Denis it's very helpful and i hope really helps to solve data loading
> performance issues, but what about increased memory usage: ~6 GB became
> 27,4 GB in off-heap cache storage or all we can do in this case is to
> increase physical memory capacity?
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Slow-data-loading-and-high-very-memory-usage-issues-tp4798p4818.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Slow data loading and high(very) memory usage issues
Posted by oskoblya <os...@at-consulting.ru>.
Thank's Denis it's very helpful and i hope really helps to solve data loading
performance issues, but what about increased memory usage: ~6 GB became
27,4 GB in off-heap cache storage or all we can do in this case is to
increase physical memory capacity?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Slow-data-loading-and-high-very-memory-usage-issues-tp4798p4818.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Slow data loading and high(very) memory usage issues
Posted by Denis Magda <dm...@gridgain.com>.
Hi,
What I see from the “top” screenshot is that 64 GB of RAM is already occupied and Linux kernel starts swapping data on disk since allowed virtual memory size is more than 90 GB. All these page ins and page outs can dramatically decrease the performance and seems that kswapd daemon works extensively (kswapd consumes around 54% of CPU cores). If you run instat or dstat you should see that the disk is became a hotspot.
In general you should avoid situations when Linux kernel starts swapping pages on the disk. As a general recommendations please do the following:
- it’s not practical to set Java heap size to 40 GB as you did because it will eventually lead to long GC pauses since neither CMS nor G1 will be able to keep up with such huge heaps under the load. My suggestion is to decrease the heap size to a value not bigger than 10 GB. Refer to this [1] JVM tuning recommendations for off-heap caches;
- tune Linux kernel to avoid long GC pauses that can be caused by Linux GC “attacks” [2]
- if all cache data will not physically fit in RAM on one machine then you have to start one more Ignite node on a separate physical machine, otherwise the swapping is unavoidable.
[1] https://apacheignite.readme.io/docs/jvm-and-system-tuning#jvm-tuning-for-clusters-with-off_heap-caches <https://apacheignite.readme.io/docs/jvm-and-system-tuning#jvm-tuning-for-clusters-with-off_heap-caches>
[2] https://apacheignite.readme.io/docs/jvm-and-system-tuning#gc-attacks-by-linux
Regards,
Denis
> On May 5, 2016, at 7:17 PM, oskoblya <os...@at-consulting.ru> wrote:
>
> Hi, I’m trying to load tSV file size ~six gigabytes (contains ~90 million
> records) into single Apache Ignite node with allocated heap size = 40
> gigabytes. For loading used IgniteDataStreamer. Code and sample (10
> records) of data is here:
> https://github.com/olegskoblya/traffic-loader.git
> Physical machine has 64 GB of RAM, 8 CPU cores (each has 2,25 GHz). Cache
> configured for storing data in unlimited off-heap memory
> (trafficCacheConf.setMemoryMode(CacheMemoryMode.OFFHEAP_TIERED);
> trafficCacheConf.setOffHeapMaxMemory(0);)
> After ~18 min physical memory usage increased to 90% and speed of loading
> catastrophically slow down (see picture below – GGVisor used in trial/demo
> mode with Apache Ignite nodes deployed on single host :) ):
> <http://apache-ignite-users.70518.x6.nabble.com/file/n4798/physical_memory_loading_near_end.png>
> <http://apache-ignite-users.70518.x6.nabble.com/file/n4798/loading_process.png>
> After manually calling GC – nothing changed (in loading speed and memory
> usage). Finally file has been loaded but last ~2 million records being
> processed the same time as ~88 previous (of course I can’t run sql query on
> loaded data because indexes was turned off for better loading performance).
> On tab SQL Viewer in column named “Total Memory” placed value 24,7GB for
> used Off Heap memory (I mean it is my ~6GB of data “eat” so many?).
> <http://apache-ignite-users.70518.x6.nabble.com/file/n4798/off_heap_used.png>
> So, it is actually normall that ~6GB of data increases in size four times
> after loading into the cache? How being used rest of memory (as 24,7 it is
> less then half of 90% used physical memory)? Is it possible to reduce such
> memory consuming? And loading 6GB of data to in-memory cache on same machine
> in 34 minutes seen as unsatisfactorily.
>
> I will be grateful for any suggestions.
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Slow-data-loading-and-high-very-memory-usage-issues-tp4798.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.