how to use memory efficiently

[Shawn Du <sh...@neulion.com.cn>]

Hi,

 

Suppose we will cache many entries(millions to billions) in ignite and cache
kept with binary.

How to decrease ignite memory usage? 

1)       Shorten cache name. I don't think it does matter, but just want
confirmed. For we use longer(more than 200 chars) cache name.

2)       Shorten cache key. I think it will. We are using md5 hash.

3)       Make column name shorter. Please confirm this. In RDMBS, it will
has no effect. 

Also shorten the column name will bring some side effects, like code
complexity and readability etc.

4)       Using off_heap. I don't know, I remembered somebody said ignite
will compress data when using off heap.

 

Anything else?

 

Thanks

Shawn

Re: 答复: how to use memory efficiently

[vdpyatkov <vl...@gmail.com>]

I do not see an memory issues at the model, but you should always remember
about cache overhead[1]
Size of any entry should be bigger than the 200 bytes.

[1]:
https://apacheignite.readme.io/docs/capacity-planning-bak#calculating-memory-usage



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/how-to-use-memory-efficiently-tp9553p9625.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

答复: how to use memory efficiently

[Shawn Du <sh...@neulion.com.cn>]

Thanks.

I will test copyOnRead. In fact, we do have troubles in using ignite. For we
think it uses too much memory. We are finding ways to decrease the memory
consumption. 
I have an idea, but I don't know it is worthy of trying or not. Please
advise.

Suppose we have several tables. These tables store some metric data. Each
table row contains several dimensions data and metrics.
These table share some common dimensions. 

Currently, we store data like this, each table has all data. And when select
no need join:
dim_1, ...,dim_x, shared_dim_1,...,shared_dim_n, metric_1,...,metric_m

we are going to store like this, we use a separated table to store shared
dimensions data:
shared_id,shared_dim_1,...,shared_dim_n
and each table use a foreign key (shared_id)to refer to the shared table:
dim_1,...,dim_x, shared_id,metric_1,...,metric_m

This is common ways in RDBMS table design, I don't know whether it works on
ignite or not. 
Also I concern the big overhead each row entry will do can save the memory.

Thanks
Shawn

-----邮件原件-----
发件人: vdpyatkov [mailto:vldpyatkov@gmail.com] 
发送时间: 2016年12月15日 21:36
收件人: user@ignite.apache.org
主题: Re: how to use memory efficiently

Hi,

1) No cache name does not affect the memory utilization.
2) Yes, the key can
3) Ignite stores the description of the class only once, and organizes hash
each object. class or field name length does not affect the memory.
4) Off_heap may be cheaper by memory consume, because in some case two copy
of entries will stored on heap (on_heap - memory mode)[1]. But ONHEAP_TIERED
does not compress data.

If you do not going to modify data, which gets from cache, set copyOnRead
flag to false:

 <bean class="org.apache.ignite.configuration.CacheConfiguration">
        <property name="copyOnRead" value="false"/>

Also, you can try to compress data (before to will put it to cache), but
this restricts your SLQ.

[1]: https://issues.apache.org/jira/browse/IGNITE-2417 



--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/how-to-use-memory-efficiently
-tp9553p9557.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: how to use memory efficiently

[vdpyatkov <vl...@gmail.com>]

Hi,

1) No cache name does not affect the memory utilization.
2) Yes, the key can
3) Ignite stores the description of the class only once, and organizes hash
each object. class or field name length does not affect the memory.
4) Off_heap may be cheaper by memory consume, because in some case two copy
of entries will stored on heap (on_heap - memory mode)[1]. But ONHEAP_TIERED
does not compress data.

If you do not going to modify data, which gets from cache, set copyOnRead
flag to false:

 <bean class="org.apache.ignite.configuration.CacheConfiguration">
        <property name="copyOnRead" value="false"/>

Also, you can try to compress data (before to will put it to cache), but
this restricts your SLQ.

[1]: https://issues.apache.org/jira/browse/IGNITE-2417 



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/how-to-use-memory-efficiently-tp9553p9557.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.