You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by qwertywx <qw...@hotmail.com> on 2018/01/17 11:11:49 UTC
Ignite loadCache performance issue
I have a 3rd party Mysql table that will be cached partially by Ignite
cluster. So on ignite, I have done this conf:
<bean id="grid.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="multicastGroup" value="228.10.10.157" />
</bean>
</property>
</bean>
</property>
<property name="clientMode" value="true" />
<property name="cacheConfiguration">
<list>
<bean
class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="crmdbcache" />
<property name="onheapCacheEnabled" value="true" />
<property name="readThrough" value="true" />
<property name="readFromBackup" value="false" />
<property name="evictionPolicy">
<bean
class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy">
<property name="maxSize" value="100" />
</bean>
</property>
<property name="indexedTypes">
<list>
<value>java.lang.Long</value>
<value>com.example.demo.UserSegment</value>
</list>
</property>
<property name="cacheStoreFactory">
<bean
class="org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory">
<property name="dataSource" ref="datasource" />
<property name="dialect">
<bean
class="org.apache.ignite.cache.store.jdbc.dialect.MySQLDialect"></bean>
</property>
<property name="parallelLoadCacheMinimumThreshold"
value="20"/>
<property name="types">
<list>
<bean
class="org.apache.ignite.cache.store.jdbc.JdbcType">
<property name="databaseSchema"
value="crm" />
<property name="databaseTable"
value="CRM_SEGMENT" />
<property name="cacheName"
value="crmdbcache" />
<property name="keyType"
value="java.lang.Long" />
<property name="valueType"
value="com.example.demo.UserSegment" />
<property name="keyFields">
<list>
<bean
class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg index="0"
value="-5" />
<constructor-arg index="1"
value="USER_ID" />
<constructor-arg index="2"
value="java.lang.Long" />
<constructor-arg index="3"
value="userId" />
</bean>
</list>
</property>
<property name="valueFields">
<list>
<bean
class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg index="0"
value="-5" />
<constructor-arg index="1"
value="ID" />
<constructor-arg index="2"
value="java.lang.Long" />
<constructor-arg index="3"
value="id" />
</bean>
<bean
class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg index="0"
value="-5" />
<constructor-arg index="1"
value="USER_ID" />
<constructor-arg index="2"
value="java.lang.Long" />
<constructor-arg index="3"
value="userId" />
</bean>
<bean
class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg index="0"
value="-5" />
<constructor-arg index="1"
value="TYPE_ID" />
<constructor-arg index="2"
value="java.lang.Long" />
<constructor-arg index="3"
value="type" />
</bean>
<bean
class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg index="0"
value="12" />
<constructor-arg index="1"
value="TAG" />
<constructor-arg index="2"
value="java.lang.String" />
<constructor-arg index="3"
value="tag" />
</bean>
</list>
</property>
</bean>
</list>
</property>
</bean>
</property>
</bean>
</list>
</property>
</bean>
So , I am also inspecting the load over ignitevisor. But it seems , it is
very slow. I have also run ignite.sh vith -v option but there seems no
problem.
So my question is: am I missing something or where should I start to look
for the problem?
It has written only 640 items in 1 minute. Increasing of thread count did
not effect performance at all. Database has 3.2 million records and I want
to add 100K of them in cache based on LRU. But with this performance, it
seems it will continue like forever
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite loadCache performance issue
Posted by qwertywx <qw...@hotmail.com>.
Denis Hi,
I have already did and it worked. Also the bottleneck about performance is
probably db related.
Thx so much for help
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite loadCache performance issue
Posted by Denis Mekhanikov <dm...@gmail.com>.
Hasan,
Unfortunately, there is no way to disable off-heap memory. Since Ignite 2.0
Ignite operates over memory pages, that are allocated in off-heap memory,
and accessed over Java Unsafe.
You can configure eviction policy in terms of occupied memory though. You
can say, that when there is a little space left, Ignite should start
evicting data pages. Here is documentation on this mechanism:
https://apacheignite.readme.io/v2.3/docs/evictions#section-off-heap-memory
Denis
чт, 18 янв. 2018 г. в 9:17, qwertywx <qw...@hotmail.com>:
> Denis Hi,
>
> Thx for the quick response. I have used at the older versions(1.3) so there
> was this "memoryMode" in there so I did not read the updated one. Thanks
> for
> the enlightenment.
>
> In order to manage the LRU functionality: Is there any way to disable
> off-heap(so the only memory would be on-heap and LRU works automaticly)?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Ignite loadCache performance issue
Posted by qwertywx <qw...@hotmail.com>.
Denis Hi,
Thx for the quick response. I have used at the older versions(1.3) so there
was this "memoryMode" in there so I did not read the updated one. Thanks for
the enlightenment.
In order to manage the LRU functionality: Is there any way to disable
off-heap(so the only memory would be on-heap and LRU works automaticly)?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite loadCache performance issue
Posted by Denis Mekhanikov <dm...@gmail.com>.
Hi Hasan!
Try disabling *onheapCacheEnabled* and *evictionPolicy *properties from
configuration. I think, it may have some performance impact.
Also make sure, that your database is connected to Ignite nodes via
high-throughput network, and is available from every data node.
> Database has 3.2 million records and I want to add 100K of them in cache
based on LRU.
Is this the reason why you enabled *evictionPolicy*? It actually serves a
different purpose. I limits amount of data, stored in oh-heap memory.
Here is documentation on Java heap eviction policy:
https://apacheignite.readme.io/docs/evictions#section-java-heap-cache
So, if you want to load only hot data from the external database, you have
to figure out yourself, which data is hot, and call IgniteCache.getAll()
<https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteCache.html#getAll(java.util.Set)>
on
corresponding keys. Read through should be enabled to make *getAll() *load
data from DB.
Denis
ср, 17 янв. 2018 г. в 14:11, qwertywx <qw...@hotmail.com>:
> I have a 3rd party Mysql table that will be cached partially by Ignite
> cluster. So on ignite, I have done this conf:
>
> <bean id="grid.cfg"
> class="org.apache.ignite.configuration.IgniteConfiguration">
> <property name="discoverySpi">
> <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> <property name="ipFinder">
> <bean
>
> class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
> <property name="multicastGroup" value="228.10.10.157"
> />
> </bean>
> </property>
> </bean>
> </property>
> <property name="clientMode" value="true" />
> <property name="cacheConfiguration">
> <list>
> <bean
> class="org.apache.ignite.configuration.CacheConfiguration">
> <property name="name" value="crmdbcache" />
> <property name="onheapCacheEnabled" value="true" />
> <property name="readThrough" value="true" />
> <property name="readFromBackup" value="false" />
> <property name="evictionPolicy">
> <bean
> class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy">
> <property name="maxSize" value="100" />
> </bean>
> </property>
> <property name="indexedTypes">
> <list>
> <value>java.lang.Long</value>
> <value>com.example.demo.UserSegment</value>
> </list>
> </property>
> <property name="cacheStoreFactory">
> <bean
> class="org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory">
> <property name="dataSource" ref="datasource" />
> <property name="dialect">
> <bean
> class="org.apache.ignite.cache.store.jdbc.dialect.MySQLDialect"></bean>
> </property>
> <property name="parallelLoadCacheMinimumThreshold"
> value="20"/>
> <property name="types">
> <list>
> <bean
> class="org.apache.ignite.cache.store.jdbc.JdbcType">
> <property name="databaseSchema"
> value="crm" />
> <property name="databaseTable"
> value="CRM_SEGMENT" />
> <property name="cacheName"
> value="crmdbcache" />
> <property name="keyType"
> value="java.lang.Long" />
> <property name="valueType"
> value="com.example.demo.UserSegment" />
> <property name="keyFields">
> <list>
> <bean
> class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
> <constructor-arg index="0"
> value="-5" />
> <constructor-arg index="1"
> value="USER_ID" />
> <constructor-arg index="2"
> value="java.lang.Long" />
> <constructor-arg index="3"
> value="userId" />
> </bean>
> </list>
> </property>
> <property name="valueFields">
> <list>
> <bean
> class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
> <constructor-arg index="0"
> value="-5" />
> <constructor-arg index="1"
> value="ID" />
> <constructor-arg index="2"
> value="java.lang.Long" />
> <constructor-arg index="3"
> value="id" />
> </bean>
> <bean
> class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
> <constructor-arg index="0"
> value="-5" />
> <constructor-arg index="1"
> value="USER_ID" />
> <constructor-arg index="2"
> value="java.lang.Long" />
> <constructor-arg index="3"
> value="userId" />
> </bean>
> <bean
> class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
> <constructor-arg index="0"
> value="-5" />
> <constructor-arg index="1"
> value="TYPE_ID" />
> <constructor-arg index="2"
> value="java.lang.Long" />
> <constructor-arg index="3"
> value="type" />
> </bean>
> <bean
> class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
> <constructor-arg index="0"
> value="12" />
> <constructor-arg index="1"
> value="TAG" />
> <constructor-arg index="2"
> value="java.lang.String" />
> <constructor-arg index="3"
> value="tag" />
> </bean>
> </list>
> </property>
> </bean>
> </list>
> </property>
> </bean>
> </property>
> </bean>
> </list>
> </property>
> </bean>
>
> So , I am also inspecting the load over ignitevisor. But it seems , it is
> very slow. I have also run ignite.sh vith -v option but there seems no
> problem.
>
> So my question is: am I missing something or where should I start to look
> for the problem?
>
> It has written only 640 items in 1 minute. Increasing of thread count did
> not effect performance at all. Database has 3.2 million records and I want
> to add 100K of them in cache based on LRU. But with this performance, it
> seems it will continue like forever
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>