You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by joaogoncalves1 <jo...@byside.com> on 2021/03/09 21:13:09 UTC
Cache group setting not taking effect
I am trying to take an Ignite cluster to production but so far we have hit a
lot of performance issues and are in the process of tuning configurations.
Use case: Our application uses the JdbcThinClient to create temporary tables
that hold partial results. We then join all those tables into one 'final
table' and delete all temporary tables. All tables are kept off-heap, in a
dataRegion.
When under load, Ignite starts being really slow to execute simple INSERT
and DROP TABLE statements, sometimes taking <10 seconds. We never see any
memory issues, which makes sense because most tables have a very small
number of rows. Cpu utilization also seems to not be a problem.
We do see that Ignite takes a lot of time to exchange partition information
between nodes, sometimes >4 seconds.
My setting is as follows:
Cluster: 4 nodes with 4 cores and 16GB RAM each
Version: Ignite 2.8.0
Heap: -Xms6g -Xmx6g
Persistence is disabled.
Config.xml:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean id="grid.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<value>10.35.2.21</value>
<value>10.35.2.22</value>
<value>10.35.2.23</value>
<value>10.35.2.24</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
<property name="dataStorageConfiguration">
<bean
class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="defaultDataRegionConfiguration">
<bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name" value="Default_Cache_Region"/>
<property name="initialSize" value="#{100L * 1024 * 1024}"/>
<property name="maxSize" value="#{6L * 1024 * 1024 * 1024}"/>
<property name="persistenceEnabled" value="false"/>
</bean>
</property>
</bean>
</property>
<property name="cacheConfiguration">
<list>
<bean id="cache-template-bean" abstract="true"
class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="table_cache_template*"/>
<property name="sqlSchema" value="PUBLIC"/>
<property name="dataRegionName" value="Default_Cache_Region"/>
<property name="groupName" value="default_group"/>
<property name="eventsDisabled" value="true"/>
<property name="backups" value="0"/>
<property name="expiryPolicyFactory">
<bean class="javax.cache.expiry.ModifiedExpiryPolicy"
factory-method="factoryOf">
<constructor-arg>
<bean class="javax.cache.expiry.Duration">
<constructor-arg value="MINUTES"/>
<constructor-arg value="6"/>
</bean>
</constructor-arg>
</bean>
</property>
</bean>
<bean id="cache-template-bean2" abstract="true"
class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="debug*"/>
<property name="sqlSchema" value="PUBLIC"/>
<property name="dataRegionName" value="Default_Cache_Region"/>
<property name="groupName" value="debug_group"/>
<property name="eventsDisabled" value="true"/>
<property name="backups" value="0"/>
</bean>
</list>
</property>
</bean>
</beans>
We tried to set 'groupName' to put all tables under the same cacheGroup, but
this setting appears to not be taking effect, because when I inspect the
caches with the *control.sh* script is shows that all caches (except for
system caches) have grpName=null and all have distinct grId, which matches
their cacheId. Does this setting not work with cache templates? I've also
tried setting it in the CREATE TABLE statement, but still no effect.
Would appreciate some insight into how to best configure ignite for this use
case.
Thanks in advance,
João Gonçalves
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Cache group setting not taking effect
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
You may choose to avoid destroying cache, instead just try to clean & reuse
them.
Number of partitions will likely not affect PME length because it is slow
due to its blocking nature, which forces all operations on the cluster to
finish before any new ops may be started.
Regards,
--
Ilya Kasnacheev
пт, 12 мар. 2021 г. в 00:19, joaogoncalves1 <jo...@byside.com>:
> Hi Ilya,
>
> Sorry for the delay, we have been heavy testing our cluster since we were
> able to set the cache groups.
> A difference between development and production configs was making me
> believe that cache groups setting didn't work, but it does. We have noticed
> some improvement in performance, but it's still not what we had hoped for.
>
> Meanwhile, we found that caches have 1024 partitions by default. We are
> considering setting the following in our config:
>
> <property name="affinity">
> <bean
>
> class="org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction">
> <property name="excludeNeighbors" value="true"/>
> <property name="partitions" value="64"/>
> </bean>
> </property>
>
> Could this help make PME's faster?
> Are there other settings we can tune to make cache creates/destroys not
> slow
> down the cluster as much, or overall improve performance?
> As I said before, all our data is temporary, we use Ignite to store some
> results for 5 minutes and then delete them. Losing this data is not a
> problem, so if there are other safety measures we can turn off that will
> help us get as fast as Ignite can go, please point them out.
>
> Best regards,
> João Gonçalves
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Cache group setting not taking effect
Posted by joaogoncalves1 <jo...@byside.com>.
Hi Ilya,
Sorry for the delay, we have been heavy testing our cluster since we were
able to set the cache groups.
A difference between development and production configs was making me
believe that cache groups setting didn't work, but it does. We have noticed
some improvement in performance, but it's still not what we had hoped for.
Meanwhile, we found that caches have 1024 partitions by default. We are
considering setting the following in our config:
<property name="affinity">
<bean
class="org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction">
<property name="excludeNeighbors" value="true"/>
<property name="partitions" value="64"/>
</bean>
</property>
Could this help make PME's faster?
Are there other settings we can tune to make cache creates/destroys not slow
down the cluster as much, or overall improve performance?
As I said before, all our data is temporary, we use Ignite to store some
results for 5 minutes and then delete them. Losing this data is not a
problem, so if there are other safety measures we can turn off that will
help us get as fast as Ignite can go, please point them out.
Best regards,
João Gonçalves
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Cache group setting not taking effect
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Are you sure that you have recreated your tables after setting group name
in their configuration? Do you have a reproducer code where you
specify grpName on a fresh cluster but it is not applied?
PME on cache create/destroy operation is blocking: it has to wait for all
operations to stop, only then may new operations start.
This would explain why you see multi-second delays.
Regards,
--
Ilya Kasnacheev
ср, 10 мар. 2021 г. в 00:13, joaogoncalves1 <jo...@byside.com>:
> I am trying to take an Ignite cluster to production but so far we have hit
> a
> lot of performance issues and are in the process of tuning configurations.
>
> Use case: Our application uses the JdbcThinClient to create temporary
> tables
> that hold partial results. We then join all those tables into one 'final
> table' and delete all temporary tables. All tables are kept off-heap, in a
> dataRegion.
>
> When under load, Ignite starts being really slow to execute simple INSERT
> and DROP TABLE statements, sometimes taking <10 seconds. We never see any
> memory issues, which makes sense because most tables have a very small
> number of rows. Cpu utilization also seems to not be a problem.
> We do see that Ignite takes a lot of time to exchange partition information
> between nodes, sometimes >4 seconds.
>
> My setting is as follows:
>
> Cluster: 4 nodes with 4 cores and 16GB RAM each
> Version: Ignite 2.8.0
> Heap: -Xms6g -Xmx6g
> Persistence is disabled.
>
> Config.xml:
>
> <?xml version="1.0" encoding="UTF-8"?>
> <beans xmlns="http://www.springframework.org/schema/beans"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="
> http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd">
>
> <bean id="grid.cfg"
> class="org.apache.ignite.configuration.IgniteConfiguration">
>
> <property name="discoverySpi">
> <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> <property name="ipFinder">
> <bean
>
> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> <property name="addresses">
> <list>
>
> <value>10.35.2.21</value>
> <value>10.35.2.22</value>
> <value>10.35.2.23</value>
> <value>10.35.2.24</value>
> </list>
> </property>
> </bean>
> </property>
> </bean>
> </property>
>
>
> <property name="dataStorageConfiguration">
> <bean
> class="org.apache.ignite.configuration.DataStorageConfiguration">
> <property name="defaultDataRegionConfiguration">
> <bean
> class="org.apache.ignite.configuration.DataRegionConfiguration">
> <property name="name" value="Default_Cache_Region"/>
> <property name="initialSize" value="#{100L * 1024 * 1024}"/>
> <property name="maxSize" value="#{6L * 1024 * 1024 * 1024}"/>
> <property name="persistenceEnabled" value="false"/>
> </bean>
> </property>
> </bean>
> </property>
>
> <property name="cacheConfiguration">
> <list>
>
> <bean id="cache-template-bean" abstract="true"
> class="org.apache.ignite.configuration.CacheConfiguration">
> <property name="name" value="table_cache_template*"/>
> <property name="sqlSchema" value="PUBLIC"/>
> <property name="dataRegionName" value="Default_Cache_Region"/>
> <property name="groupName" value="default_group"/>
> <property name="eventsDisabled" value="true"/>
> <property name="backups" value="0"/>
> <property name="expiryPolicyFactory">
> <bean class="javax.cache.expiry.ModifiedExpiryPolicy"
> factory-method="factoryOf">
> <constructor-arg>
> <bean class="javax.cache.expiry.Duration">
> <constructor-arg value="MINUTES"/>
> <constructor-arg value="6"/>
> </bean>
> </constructor-arg>
> </bean>
> </property>
> </bean>
>
>
> <bean id="cache-template-bean2" abstract="true"
> class="org.apache.ignite.configuration.CacheConfiguration">
> <property name="name" value="debug*"/>
> <property name="sqlSchema" value="PUBLIC"/>
> <property name="dataRegionName" value="Default_Cache_Region"/>
> <property name="groupName" value="debug_group"/>
> <property name="eventsDisabled" value="true"/>
> <property name="backups" value="0"/>
> </bean>
>
> </list>
> </property>
> </bean>
> </beans>
>
> We tried to set 'groupName' to put all tables under the same cacheGroup,
> but
> this setting appears to not be taking effect, because when I inspect the
> caches with the *control.sh* script is shows that all caches (except for
> system caches) have grpName=null and all have distinct grId, which matches
> their cacheId. Does this setting not work with cache templates? I've also
> tried setting it in the CREATE TABLE statement, but still no effect.
>
> Would appreciate some insight into how to best configure ignite for this
> use
> case.
>
> Thanks in advance,
> João Gonçalves
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>