You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by "krkumar24061975@gmail.com" <kr...@gmail.com> on 2020/04/07 12:33:19 UTC
Ignite Restart takes long time
Hi Guys - I have this problem that's very recent and nothing significant
changed in the system but all of a sudden still takes lot of time to
restart. I do a graceful shutdown of one of the nodes and restart the node
after some updates. Now the ignite takes almost 30 minutes to initiate and
that's killing us.
Here is the snippet of my ignite configuration:
<bean id="igniteContainer" class="org.apache.ignite.IgniteSpringBean"
depends-on="dataSource">
<property name="configuration">
<bean id="ignite.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="workDirectory" value="${work.space}"></property>
<property name="communicationSpi">
<bean
class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
<property name="connectTimeout" value="10000"></property>
<property name="socketWriteTimeout" value="30000"></property>
</bean>
</property>
<property name="failureDetectionTimeout" value="120000"></property>
<property name="rebalanceThreadPoolSize" value="8"></property>
<property name="publicThreadPoolSize" value="64" />
<property name="systemThreadPoolSize" value="32" />
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="writeThrottlingEnabled" value="true"></property>
<property name="defaultDataRegionConfiguration">
<bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="initialSize" value="#{512L * 1024 *
1024}"></property>
<property name="maxSize" value="#{20L * 1024 * 1024 * 1024}" />
<property name="persistenceEnabled" value="true" />
</bean>
</property>
<property name="dataRegionConfigurations">
<list>
<bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name" value="indexMainDataRegion" />
<property name="initialSize" value="#{512L * 1024 * 1024}" />
<property name="maxSize" value="#{20L * 1024 * 1024 * 1024}" />
<property name="persistenceEnabled" value="true" />
</bean>
<bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name" value="indexDtlDataRegion" />
<property name="initialSize" value="#{512L * 1024 * 1024}" />
<property name="maxSize" value="#{20L * 1024 * 1024 * 1024}" />
<property name="persistenceEnabled" value="true" />
</bean>
</list>
</property>
<property name="pageSize" value="#{4 * 1024}" />
<property name="storagePath" value="${grid.data}" />
<property name="walPath" value="${grid.wal}" />
<property name="walArchivePath" value="${grid.wal}" />
<property name="walMode" value="BACKGROUND" />
<property name="walFlushFrequency" value="10000"></property>
</bean>
</property>
Here is the code snippet of how we create the cache
CacheConfiguration<Long, byte[]> cacheConfig = new CacheConfiguration<>();
cacheConfig.setCacheMode(CacheMode.PARTITIONED);
cacheConfig.setRebalanceMode(CacheRebalanceMode.ASYNC);
cacheConfig.setRebalanceDelay(60000);
cacheConfig.setName("eventCache-" + xxx + "-" + xxx);
cacheConfig.setBackups(1);
cacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
cacheConfig.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
We start the ignite like the following
context = new ClassPathXmlApplicationContext("xxx.xml");
logger.info("Cluster kernel is initializing ...Done");
engine = (Ignite) context.getBean("igniteContainer");
engine.cluster().active(true);
Thanx and Regards,
KR Kumar
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite Restart takes long time
Posted by Stephen Darlington <st...@gridgain.com>.
I don’t think you should be activating the cluster every time you restart a node. Generally it’s best to activate the cluster once and manage the baseline topology “manually” after that.
> On 7 Apr 2020, at 17:20, krkumar24061975@gmail.com wrote:
>
> Hi Gianluca Bonetti - Thanks for the help. I actually have that JVM parameter
> and now I have removed that. Looks like its working and I will do few more
> rounds of testing and then update you.
>
> Also any idea why ignite.active(true) taking almost 5 minutes whenever i
> restart the node?
>
> Again a big thanks,
>
> KR Kumar
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite Restart takes long time
Posted by "krkumar24061975@gmail.com" <kr...@gmail.com>.
Hi Gianluca Bonetti - Initially if I restart when the data is around 100
million entries, it takes about 3 minutes, now I am testing it with 270
million and it takes about 12 minutes. Upon looking into the ignite code,
this is where its taking time in the class
org.apache.ignite.internal.IgntionEx where its trying to start the cluster:
try {
IgniteKernal grid0 = new
IgniteKernal(startCtx.springContext());
// Init here to make grid available to lifecycle listeners.
grid = grid0;
grid0.start(
myCfg,
utilityCacheExecSvc,
execSvc,
svcExecSvc,
sysExecSvc,
stripedExecSvc,
p2pExecSvc,
mgmtExecSvc,
igfsExecSvc,
dataStreamerExecSvc,
restExecSvc,
affExecSvc,
idxExecSvc,
callbackExecSvc,
qryExecSvc,
schemaExecSvc,
customExecSvcs,
new CA() {
@Override public void apply() {
startLatch.countDown();
}
},
workerRegistry,
oomeHnd
);
Thanx and Regards,
KR Kumar
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite Restart takes long time
Posted by Gianluca Bonetti <gi...@gmail.com>.
Hello Krumar
Full explanation about the slowness when using AlwaysPreTouch comes from
this article https://access.redhat.com/solutions/2685771
Is it still taking 5 minutes to start, after removing the AlwaysPreTouch
parameter?
Cheers
Gianluca
Il giorno mar 7 apr 2020 alle ore 18:20 krkumar24061975@gmail.com <
krkumar24061975@gmail.com> ha scritto:
> Hi Gianluca Bonetti - Thanks for the help. I actually have that JVM
> parameter
> and now I have removed that. Looks like its working and I will do few more
> rounds of testing and then update you.
>
> Also any idea why ignite.active(true) taking almost 5 minutes whenever i
> restart the node?
>
> Again a big thanks,
>
> KR Kumar
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Ignite Restart takes long time
Posted by "krkumar24061975@gmail.com" <kr...@gmail.com>.
Hi Gianluca Bonetti - Thanks for the help. I actually have that JVM parameter
and now I have removed that. Looks like its working and I will do few more
rounds of testing and then update you.
Also any idea why ignite.active(true) taking almost 5 minutes whenever i
restart the node?
Again a big thanks,
KR Kumar
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite Restart takes long time
Posted by Gianluca Bonetti <gi...@gmail.com>.
Hello Kumar
It seems a problem similar to the one I had in the past.
Please check that you are running the JVM *without* the
"-XX:+AlwaysPreTouch" parameter.
Also, in my experience, using /dev/random as entropy source could block
startup for long time, hence I prefer using /dev/urandom
(set securerandom.source=file:/dev/random in
/etc/java-8-openjdk/security/java.security for Debian-like systems)
Cheers
Gianluca
Il giorno mar 7 apr 2020 alle ore 14:33 krkumar24061975@gmail.com <
krkumar24061975@gmail.com> ha scritto:
> Hi Guys - I have this problem that's very recent and nothing significant
> changed in the system but all of a sudden still takes lot of time to
> restart. I do a graceful shutdown of one of the nodes and restart the node
> after some updates. Now the ignite takes almost 30 minutes to initiate and
> that's killing us.
>
> Here is the snippet of my ignite configuration:
>
> <bean id="igniteContainer" class="org.apache.ignite.IgniteSpringBean"
> depends-on="dataSource">
> <property name="configuration">
> <bean id="ignite.cfg"
> class="org.apache.ignite.configuration.IgniteConfiguration">
> <property name="workDirectory" value="${
> work.space}"></property>
> <property name="communicationSpi">
> <bean
> class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
> <property
> name="connectTimeout" value="10000"></property>
> <property
> name="socketWriteTimeout" value="30000"></property>
> </bean>
> </property>
> <property name="failureDetectionTimeout"
> value="120000"></property>
> <property name="rebalanceThreadPoolSize"
> value="8"></property>
> <property name="publicThreadPoolSize"
> value="64" />
> <property name="systemThreadPoolSize"
> value="32" />
> <property name="dataStorageConfiguration">
> <bean
> class="org.apache.ignite.configuration.DataStorageConfiguration">
> <property
> name="writeThrottlingEnabled" value="true"></property>
> <property
> name="defaultDataRegionConfiguration">
> <bean
> class="org.apache.ignite.configuration.DataRegionConfiguration">
> <property
> name="initialSize" value="#{512L * 1024 *
> 1024}"></property>
> <property
> name="maxSize" value="#{20L * 1024 * 1024 * 1024}" />
> <property
> name="persistenceEnabled" value="true" />
> </bean>
> </property>
> <property
> name="dataRegionConfigurations">
> <list>
> <bean
> class="org.apache.ignite.configuration.DataRegionConfiguration">
>
> <property name="name" value="indexMainDataRegion" />
>
> <property name="initialSize" value="#{512L * 1024 * 1024}" />
>
> <property name="maxSize" value="#{20L * 1024 * 1024 * 1024}" />
>
> <property name="persistenceEnabled" value="true" />
> </bean>
> <bean
> class="org.apache.ignite.configuration.DataRegionConfiguration">
>
> <property name="name" value="indexDtlDataRegion" />
>
> <property name="initialSize" value="#{512L * 1024 * 1024}" />
>
> <property name="maxSize" value="#{20L * 1024 * 1024 * 1024}" />
>
> <property name="persistenceEnabled" value="true" />
> </bean>
> </list>
> </property>
> <property name="pageSize"
> value="#{4 * 1024}" />
> <property
> name="storagePath" value="${grid.data}" />
> <property name="walPath"
> value="${grid.wal}" />
> <property
> name="walArchivePath" value="${grid.wal}" />
> <property name="walMode"
> value="BACKGROUND" />
> <property
> name="walFlushFrequency" value="10000"></property>
> </bean>
> </property>
>
> Here is the code snippet of how we create the cache
>
> CacheConfiguration<Long, byte[]> cacheConfig = new
> CacheConfiguration<>();
> cacheConfig.setCacheMode(CacheMode.PARTITIONED);
> cacheConfig.setRebalanceMode(CacheRebalanceMode.ASYNC);
> cacheConfig.setRebalanceDelay(60000);
> cacheConfig.setName("eventCache-" + xxx + "-" + xxx);
> cacheConfig.setBackups(1);
> cacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
>
>
> cacheConfig.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
>
> We start the ignite like the following
>
> context = new ClassPathXmlApplicationContext("xxx.xml");
>
> logger.info("Cluster kernel is initializing ...Done");
> engine = (Ignite) context.getBean("igniteContainer");
> engine.cluster().active(true);
>
>
> Thanx and Regards,
> KR Kumar
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>