Reducing On-Heap Cache Memory Footprint

[abarbaro <ab...@powercosts.com>]

Hello,

I'm trying to understand the memory footprint for ON_HEAP caches in Ignite
with the goal of sizing server nodes properly.  

I have set up a simple cache test with a single server node and 2,000,000
entries of type <Long, PortfolioAsset> and collected a Heap Dump.  Each
instance of PortfolioAsset is 32 bytes, and Long is 24 bytes.  

When looking at the Heap Histogram and seeing the amount of memory overhead
I would like to know if there if there is a way to reduce/eliminate it.  In
picture shown below I can account for the rows in green:  my keys, my
values, the serialized version of the values (BinaryObjectImpl), and some
overhead for the cache entries.  This part makes total sense to me. 
However, I can't quite make sense of the 3 entries in red.  

<http://apache-ignite-users.70518.x6.nabble.com/file/n3836/Apache_Ignite_Memory_Footprint.png> 

How can I tweak my cache/ignite configuration parameters to remove this
overhead?  The data and entries only take 260 MB while the other stuff takes
380 MB.  Here's my config:

		<property name="cacheConfiguration">
			<bean class="org.apache.ignite.configuration.CacheConfiguration">
				<property name="memoryMode" value="ONHEAP_TIERED"/> 
				<property name="atomicityMode" value="ATOMIC"/>
				<property name="cacheMode" value="PARTITIONED"/>
				<property name="offHeapMaxMemory" value="-1"/>
				<property name="copyOnRead" value="false"/>
				<property name="backups" value="0"/>
				<property name="eagerTtl" value="false"/> 
				<property name="startSize" value="10000"/> 
				<property name="affinity">
					<bean
class="org.apache.ignite.cache.affinity.fair.FairAffinityFunction">
						<property name="partitions" value="32"/>
					</bean>
				</property>
			</bean>
		</property>



Thanks in advance for your help.


Andres



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Reducing-On-Heap-Cache-Memory-Footprint-tp3836.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Reducing On-Heap Cache Memory Footprint

[Artem Shutak <as...@gridgain.com>]

Hello,

I would suggest you to follow the following algorithm when calculating
memory usage by Ignite.

CALCULATING THE MEMORY USAGE
1. Calculate primary data size: multiply the size of one entry in bytes by
the total number of entries. Note, that Ignite will typically add around 200
bytes overhead to each entry.
2. If you have backups, multiply by their number
3. Indexes also require memory. Basic use cases will add a 30% increase
4. Add around 20MB per cache. This value can be reduced if to decrease
partitions count and/or
IgniteSystemProperties.IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE
5. Add around 200-300MB per node for internal memory and reasonable amount
of memory for JVM and GC to operate efficiently

So, in your case with 1 node and 0 backups it is:

(24+32+200)*2_000_000 bytes + 20 mb + 200 mb > 700 mb.

Thanks,
- Artem -



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Reducing-On-Heap-Cache-Memory-Footprint-tp3836p3909.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.