You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Prashant312 <p....@crif.com> on 2017/06/30 09:46:12 UTC
Apache heap getting increase with cacheConfig.setCopyOnRead as
false
Hi,
I just have a simple java code from one code I am putting data in ignite
cache and from the other code I am trying to get the value. But I have
observed that the heap size is getting doubled when ever we are getting data
from cache.
i have set setCopyOnRead property as false.
Thanks,
Prashant Verma
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Apache-heap-getting-increase-with-cacheConfig-setCopyOnRead-as-false-tp14175.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Apache heap getting increase with cacheConfig.setCopyOnRead as
false
Posted by vkulichenko <va...@gmail.com>.
Prashant,
This is maximum heap size controlled by JVM Xmx property. I guess you just
set the same for both server and client.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Apache-heap-getting-increase-with-cacheConfig-setCopyOnRead-as-false-tp14175p14348.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Apache heap getting increase with cacheConfig.setCopyOnRead as
false
Posted by Prashant312 <p....@crif.com>.
Hi,
Yes currently we have client and server on the same server.
I agree that some memory might consume when a client gets connected to the
cluster, but here in my example the memory is getting exactly doubled and
the same is being observed in production, where the cache size is 20 Gb and
when we increase the clients it is getting 40 Gb, 60 Gb and so on.
[10:37:28] Topology snapshot [ver=1, servers=1, clients=0, CPUs=4,
heap=1.8GB]
Done....cache
[10:37:46] Topology snapshot [ver=2, servers=1, clients=1, CPUs=4,
heap=3.5GB]
[10:37:48] Topology snapshot [ver=3, servers=1, clients=0, CPUs=4,
heap=1.8GB]
Thanks,
Prashant
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Apache-heap-getting-increase-with-cacheConfig-setCopyOnRead-as-false-tp14175p14300.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Apache heap getting increase with cacheConfig.setCopyOnRead as
false
Posted by vkulichenko <va...@gmail.com>.
Prashant,
Do you mean that you have a client and a server on the same box? If so, why
do you need that? Client node is also a JVM process and it will always
consume some memory. Of course, much less than server as it doesn't store
data.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Apache-heap-getting-increase-with-cacheConfig-setCopyOnRead-as-false-tp14175p14286.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Apache heap getting increase with cacheConfig.setCopyOnRead as
false
Posted by Prashant312 <p....@crif.com>.
Hi,
Please find below my code for creating and populating cache.
----
package ignite;
import java.util.Arrays;
import javax.cache.configuration.Factory;
import org.apache.ignite.Ignite;
import org.apache.ignite.IgniteCache;
import org.apache.ignite.IgniteSystemProperties;
import org.apache.ignite.Ignition;
import org.apache.ignite.cache.CacheAtomicityMode;
import org.apache.ignite.cache.CacheMode;
import org.apache.ignite.configuration.CacheConfiguration;
import org.apache.ignite.configuration.IgniteConfiguration;
import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
import
org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
public class PopulateCache {
public static IgniteCache<String, String> igniteCache=null;
public static void main(String[] args) {
Ignition.setClientMode(false);
System.setProperty(IgniteSystemProperties.IGNITE_QUIET,"false");
System.setProperty("java.net.preferIPv4Stack" , "true");
TcpDiscoverySpi spi = new TcpDiscoverySpi();
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
ipFinder.setAddresses(Arrays.asList("localhost",
"localhost:47500..47509"));
spi.setIpFinder(ipFinder);
IgniteConfiguration cfg = new IgniteConfiguration();
cfg.setDiscoverySpi(spi);
Ignite ignite = Ignition.start(cfg);
CacheConfiguration<String, String> cacheConfig=new
CacheConfiguration<String, String>();
cacheConfig.setBackups(1);
cacheConfig.setCopyOnRead(false);
cacheConfig.setName("testCache");
cacheConfig.setCacheMode(CacheMode.PARTITIONED);
igniteCache=ignite.getOrCreateCache(cacheConfig);
igniteCache.put("100", "abc");
System.out.println("Done....cache");
}
public static Object getCacheData(String key){
if(igniteCache!=null){
return igniteCache.get(key);
}
return key;
}
}
-------------------------------------------------------
Log -
/
You can find that during creation of cache where we are putting just one
value heap is 1.8 Gb.
Topology snapshot [ver=1, servers=1, clients=0, CPUs=4, heap=1.8GB]
/
Now code for getting the cache is below
------------------------------------------------------
package ignite;
import java.util.Arrays;
import java.util.Properties;
import org.apache.ignite.Ignite;
import org.apache.ignite.IgniteCache;
import org.apache.ignite.IgniteSystemProperties;
import org.apache.ignite.Ignition;
import org.apache.ignite.cache.CacheAtomicityMode;
import org.apache.ignite.cache.CacheMode;
import org.apache.ignite.configuration.CacheConfiguration;
import org.apache.ignite.configuration.IgniteConfiguration;
import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
import
org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
import static
org.apache.ignite.IgniteSystemProperties.IGNITE_REST_START_ON_CLIENT;
public class StartIgnite {
public static IgniteCache<String, String> igniteCache=null;
public static void startIgnite(){
Ignition.setClientMode(true);
System.setProperty(IgniteSystemProperties.IGNITE_QUIET,"false");
TcpDiscoverySpi spi = new TcpDiscoverySpi();
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
ipFinder.setAddresses(Arrays.asList("localhost",
"localhost:47500..47509"));
spi.setIpFinder(ipFinder);
System.out.println(ipFinder);
IgniteConfiguration cfg = new IgniteConfiguration();
cfg.setDiscoverySpi(spi);
Ignite ignite = Ignition.start(cfg);
CacheConfiguration<String, String> cacheConfig=new
CacheConfiguration<String, String>();
cacheConfig.setName("testCache");
cacheConfig.setCopyOnRead(false);
//cacheConfig.
//cacheConfig.s
cacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
igniteCache=ignite.getOrCreateCache(cacheConfig);
System.out.println("cache name is "+ igniteCache.getName());
}
public static void stopIgnite(){
Ignition.stop(true);
}
public static String getCacheData(String id){
return igniteCache.get(id);
}
public static void main(String[] args) {
startIgnite();
//System.out.println(Cache.getCacheData(CacheConstants.CLEANSING_TOKENS));
String obj=igniteCache.get("200");
String obj2=igniteCache.get("100");
System.out.println(obj);
System.out.println(obj2);
stopIgnite();
}
}
--------------------------------------------------
When we are executing above code heap gets increased to 3.5 GB.
09:22:55] Topology snapshot [ver=2, servers=1, clients=1, CPUs=4,
heap=3.5GB]
[09:22:57] Topology snapshot [ver=3, servers=1, clients=0, CPUs=4,
heap=1.8GB]
We want to avoid increase in heap size, in production we have cache of
around 30 Gb and we are running the process parallel, causing heap size
full.
Thanks,
Prashant
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Apache-heap-getting-increase-with-cacheConfig-setCopyOnRead-as-false-tp14175p14228.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Apache heap getting increase with cacheConfig.setCopyOnRead as
false
Posted by vkulichenko <va...@gmail.com>.
Prashant,
To avoid what? Why did you set copyOnRead to false in the first place?
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Apache-heap-getting-increase-with-cacheConfig-setCopyOnRead-as-false-tp14175p14190.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Apache heap getting increase with cacheConfig.setCopyOnRead as false
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi Prashant,
If you cache objects with a lot of fields and you need to get just somse of
them. Please, take a look a binary object concept [1].
You can use binary objects representations with cache projection
IgniteCache.withKeepBinary() to avoid unnecessary object deserialization.
[1] https://apacheignite.readme.io/docs/binary-marshaller#basic-concepts
On Fri, Jun 30, 2017 at 10:27 PM, Verma Prashant <P....@crif.com> wrote:
> Are there any other ways to avoid this. We have 20 gb of cache and when we
> are fetching the data cache from 9 different mappers of single job memory
> utilization becomes 180 gb.
>
> Thanks,
> Prashant Verma
>
>
>
> Sent from my Samsung Galaxy smartphone.
>
>
> -------- Original message --------
> From: vkulichenko <va...@gmail.com>
> Date: 30/06/2017 23:41 (GMT+05:30)
> To: user@ignite.apache.org
> Subject: Re: Apache heap getting increase with cacheConfig.setCopyOnRead
> as false
>
> Prashant,
>
> Ignite always stores binary representation of an object. Whenever you read
> it, its is deserialized which actually creates a copy. In case copyOnRead
> is
> set to false, deserialized object if also cached so that deserialization
> step is skipped. But this doubles memory consumption.
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Apache-heap-getting-increase-with-
> cacheConfig-setCopyOnRead-as-false-tp14175p14184.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
>
--
Best regards,
Andrey V. Mashenkov
Re: Apache heap getting increase with cacheConfig.setCopyOnRead as
false
Posted by Verma Prashant <P....@crif.com>.
Are there any other ways to avoid this. We have 20 gb of cache and when we are fetching the data cache from 9 different mappers of single job memory utilization becomes 180 gb.
Thanks,
Prashant Verma
Sent from my Samsung Galaxy smartphone.
-------- Original message --------
From: vkulichenko <va...@gmail.com>
Date: 30/06/2017 23:41 (GMT+05:30)
To: user@ignite.apache.org
Subject: Re: Apache heap getting increase with cacheConfig.setCopyOnRead as false
Prashant,
Ignite always stores binary representation of an object. Whenever you read
it, its is deserialized which actually creates a copy. In case copyOnRead is
set to false, deserialized object if also cached so that deserialization
step is skipped. But this doubles memory consumption.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Apache-heap-getting-increase-with-cacheConfig-setCopyOnRead-as-false-tp14175p14184.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Apache heap getting increase with cacheConfig.setCopyOnRead as
false
Posted by vkulichenko <va...@gmail.com>.
Prashant,
Ignite always stores binary representation of an object. Whenever you read
it, its is deserialized which actually creates a copy. In case copyOnRead is
set to false, deserialized object if also cached so that deserialization
step is skipped. But this doubles memory consumption.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Apache-heap-getting-increase-with-cacheConfig-setCopyOnRead-as-false-tp14175p14184.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.