You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Prasad Bhalerao <pr...@gmail.com> on 2018/06/27 00:50:20 UTC
Cache size in offheap mem in bytes
Hi,
an object takes 200 bytes on heap and cache has such 50 million objects
stored in it.
Is it ok to calculate cache size as follows?
Cache size in bytes = object size * no of objects
Cache size in bytes = 200 * 50 million
Thanks,
Prasad
Re: Cache size in offheap mem in bytes
Posted by Alex Plehanov <pl...@gmail.com>.
It's incorrect to use cache object to calculate cache data size. What you
got now is a footprint of the Ignite infrastructure used to manage your
data, but not a footprint of your data itself, since data are stored in
off-heap and this tool only calculate on-heap size of objects referenced by
cache object. As you can see in your sample 1 000 000 objects was created
with 5 longs each, but you got only 2 872 685 longs in footprint.
To roughly estimate off-heap memory needed for cache data you can enable
persistent, put large amount of entries to the cache and then measure size
of cache folder (you should also wait until checkpoint complete to get
correct results). Also page in off-heap memory have some extra header, so
final off-heap memory needed will be a little bit larger then cache folder
size.
2018-06-27 17:55 GMT+03:00 Prasad Bhalerao <pr...@gmail.com>:
> Hi,
>
> I have written a test program to find the cache size in bytes. I am using
> this <https://jitpack.io/p/methodho/memory-measurer> tool
> (memory-measurer) to find the object sizes and its foot prints. I tried to
> use this tool to find the cache size in bytes. I am using on heap cache.
>
> Am I using the correct object (cache in this case) to check the cache size
> in memory? Can some please clarify?
>
> I tried to use Jprofiler in instrumentation mode to check on-heap cache
> size, but could not locate the exact class/objects.
>
> private void pushData(CacheName cacheName, List<? extends Data<? extends DataKey>> datas) {
>
> final IgniteCache<DataKey, Data> cache = ignite.cache(cacheName.name());
>
> LOGGER.info("cache size in bytes : {}", IpV4RangeTest.bytesToHuman(MemoryMeasurer.measureBytes(cache)));
>
> TestDataObj data8 = null;
> for(int ctr=0;ctr<1_000_000;ctr++){
> data8 = new TestDataObj();
> data8.setId(ctr);
> data8.setSubscriptionId(SUBSCRIPTION_ID);
> cache.put(data8.getKey(), data8);
> }
>
> LOGGER.info("TestDataObj size in bytes : {}",
> IpV4RangeTest.bytesToHuman(MemoryMeasurer.measureBytes(data8)));
>
> LOGGER.info("cache size in bytes : {}",
> IpV4RangeTest.bytesToHuman(MemoryMeasurer.measureBytes(cache)));
>
> Footprint footprint = ObjectGraphMeasurer.measure(cache);
>
> LOGGER.info("{} Footprint={}", cacheName.name(), footprint.toString());
>
> LOGGER.info("{} size={}", cacheName.name(), cache.size());
> try {
> Thread.sleep(100000);
> } catch (InterruptedException e) {
> e.printStackTrace();
> }
> }
>
>
>
> * Output:*
>
> *cache size in bytes* : 36.4 MB *[Empty Cache]*
>
> * TestDataObj size in bytes : 64 byte cache size in bytes* : 493.23 MB
> *[After adding 1 million objects]*
> *Footprint*=Footprint
> [objects=10962325,
> references=30951850,
> primitives={byte=130277671, long=2872685, double=58, float=52015,
> int=11651118, boolean=2156105, char=1905788, short=10457}]
>
>
> *Inside TestDataObj class:*
>
> public class TestDataObj implements Data<DefaultDataAffinityKey> {
>
> @QuerySqlField
> private long id;
>
> @QuerySqlField(orderedGroups = {@QuerySqlField.Group(name = "ag_domain_nb_override_data", order = 1)})
> private long assetGroupDomainId;
>
> @QuerySqlField(orderedGroups = {@QuerySqlField.Group(name = "ag_domain_nb_override_data", order = 2)})
> private long startIp;
>
> @QuerySqlField(orderedGroups = {@QuerySqlField.Group(name = "ag_domain_nb_override_data", order = 3)})
> private long endIp;
>
> private long subscriptionId;
>
> @QuerySqlField(orderedGroups = {@QuerySqlField.Group(name = "ag_domain_nb_override_data", order = 4)})
> private int partitionId;
>
> private boolean isUpdatedData;
>
> }
>
>
>
> Thanks,
> PRasad
>
> On Wed, Jun 27, 2018 at 3:00 PM dkarachentsev <dk...@gridgain.com>
> wrote:
>
>> 1) This applicable to Ignite. As it grown from GridGain sometimes it may
>> appear in docs, because missed fro removal.
>> 2) Yes, and I would say overhead could be even bigger. But anyway I cannot
>> say definitely how much, because Ignite doesn't store data sequentially,
>> there a lot of nuances.
>> 3) Ignite transaction caches entries on-heap and this works only for
>> TRANSACTIONAL caches.
>>
>> Thanks!
>> -Dmitry
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
Re: Cache size in offheap mem in bytes
Posted by Prasad Bhalerao <pr...@gmail.com>.
Hi,
I have written a test program to find the cache size in bytes. I am using
this <https://jitpack.io/p/methodho/memory-measurer> tool (memory-measurer)
to find the object sizes and its foot prints. I tried to use this tool to
find the cache size in bytes. I am using on heap cache.
Am I using the correct object (cache in this case) to check the cache size
in memory? Can some please clarify?
I tried to use Jprofiler in instrumentation mode to check on-heap cache
size, but could not locate the exact class/objects.
private void pushData(CacheName cacheName, List<? extends Data<?
extends DataKey>> datas) {
final IgniteCache<DataKey, Data> cache = ignite.cache(cacheName.name());
LOGGER.info("cache size in bytes : {}",
IpV4RangeTest.bytesToHuman(MemoryMeasurer.measureBytes(cache)));
TestDataObj data8 = null;
for(int ctr=0;ctr<1_000_000;ctr++){
data8 = new TestDataObj();
data8.setId(ctr);
data8.setSubscriptionId(SUBSCRIPTION_ID);
cache.put(data8.getKey(), data8);
}
LOGGER.info("TestDataObj size in bytes : {}",
IpV4RangeTest.bytesToHuman(MemoryMeasurer.measureBytes(data8)));
LOGGER.info("cache size in bytes : {}",
IpV4RangeTest.bytesToHuman(MemoryMeasurer.measureBytes(cache)));
Footprint footprint = ObjectGraphMeasurer.measure(cache);
LOGGER.info("{} Footprint={}", cacheName.name(), footprint.toString());
LOGGER.info("{} size={}", cacheName.name(), cache.size());
try {
Thread.sleep(100000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
* Output:*
*cache size in bytes* : 36.4 MB *[Empty Cache]*
* TestDataObj size in bytes : 64 byte cache size in bytes* : 493.23 MB
*[After adding 1 million objects]*
*Footprint*=Footprint
[objects=10962325,
references=30951850,
primitives={byte=130277671, long=2872685, double=58, float=52015,
int=11651118, boolean=2156105, char=1905788, short=10457}]
*Inside TestDataObj class:*
public class TestDataObj implements Data<DefaultDataAffinityKey> {
@QuerySqlField
private long id;
@QuerySqlField(orderedGroups = {@QuerySqlField.Group(name =
"ag_domain_nb_override_data", order = 1)})
private long assetGroupDomainId;
@QuerySqlField(orderedGroups = {@QuerySqlField.Group(name =
"ag_domain_nb_override_data", order = 2)})
private long startIp;
@QuerySqlField(orderedGroups = {@QuerySqlField.Group(name =
"ag_domain_nb_override_data", order = 3)})
private long endIp;
private long subscriptionId;
@QuerySqlField(orderedGroups = {@QuerySqlField.Group(name =
"ag_domain_nb_override_data", order = 4)})
private int partitionId;
private boolean isUpdatedData;
}
Thanks,
PRasad
On Wed, Jun 27, 2018 at 3:00 PM dkarachentsev <dk...@gridgain.com>
wrote:
> 1) This applicable to Ignite. As it grown from GridGain sometimes it may
> appear in docs, because missed fro removal.
> 2) Yes, and I would say overhead could be even bigger. But anyway I cannot
> say definitely how much, because Ignite doesn't store data sequentially,
> there a lot of nuances.
> 3) Ignite transaction caches entries on-heap and this works only for
> TRANSACTIONAL caches.
>
> Thanks!
> -Dmitry
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Cache size in offheap mem in bytes
Posted by dkarachentsev <dk...@gridgain.com>.
1) This applicable to Ignite. As it grown from GridGain sometimes it may
appear in docs, because missed fro removal.
2) Yes, and I would say overhead could be even bigger. But anyway I cannot
say definitely how much, because Ignite doesn't store data sequentially,
there a lot of nuances.
3) Ignite transaction caches entries on-heap and this works only for
TRANSACTIONAL caches.
Thanks!
-Dmitry
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Cache size in offheap mem in bytes
Posted by Prasad Bhalerao <pr...@gmail.com>.
Hi,
Thank you for the explanation.
As per this link
https://apacheignite.readme.io/docs/capacity-planning#section-calculating-memory-usage
,
"GridGain will typically add around 200 bytes overhead to each entry."
1) Why does it specifically say GridGain will add around 200 bytes overhead
for each entry?
Does ignite not add this overhead?
3) Does this
<https://apacheignite.readme.io/docs/capacity-planning#section-memory-capacity-planning-example>
example already considers 200 bytes overhead per entry?
4) When ignite transaction is started, it keeps the cache updates in temp
cache or transactional cache and it is kept in on-heap.
Does this rule apply to transactional cache as well?
Regard,
Prasad
On Wed, Jun 27, 2018 at 2:15 PM dkarachentsev <dk...@gridgain.com>
wrote:
> Hi,
>
> It will be incorrect, because entries are not stored sequentially, there
> are
> a lot of infrastructure that requires additional space. For example, memory
> is divided into pages and each page has header, each entry has key and
> value, version and other service information. For quick access to entries
> is
> created b+tree (which consists of pages too), synchronization primitives,
> links and binary object overhead to allow access to it's fields.
>
> It's quire hard to say how much memory will be required, just approximately
> [1]. The best way will be to load some number of data and measure how much
> memory was consumed. You need to make many such measures, because only on
> large number of entries memory consumption will increase linearly.
>
> [1] https://apacheignite.readme.io/docs/capacity-planning
>
> Thanks!
> -Dmitry
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Cache size in offheap mem in bytes
Posted by dkarachentsev <dk...@gridgain.com>.
Hi,
It will be incorrect, because entries are not stored sequentially, there are
a lot of infrastructure that requires additional space. For example, memory
is divided into pages and each page has header, each entry has key and
value, version and other service information. For quick access to entries is
created b+tree (which consists of pages too), synchronization primitives,
links and binary object overhead to allow access to it's fields.
It's quire hard to say how much memory will be required, just approximately
[1]. The best way will be to load some number of data and measure how much
memory was consumed. You need to make many such measures, because only on
large number of entries memory consumption will increase linearly.
[1] https://apacheignite.readme.io/docs/capacity-planning
Thanks!
-Dmitry
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/