You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by praveeng <pr...@gmail.com> on 2018/10/22 09:53:15 UTC
Inserting data into Ignite thread is hung
Hi ,
Version: 1.8
Java: 1.7.0_45-b18 [ we can't upgrade the java version to 1.8 due to some
internal restriction.]
We are facing that the ignite thread is hung when we are trying to insert
the data.
This is not happening that much frequently.
We don't see any exceptions in ignite except below that too the below
exception has come only on one server. Can you please check and suggest us.
*Exception in ignite server:*
[02:15:14,320][WARN
][grid-nio-worker-1-#10%CasinoApacheIgniteServices%][TcpCommunicationSpi]
Failed to process selector key (will close): GridSelectorNioSessionImpl
[selectorIdx=1, queueSize=0, writeBuf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768
cap=32768], recovery=GridNioRecoveryDescriptor [acked=2060041, resendCnt=0,
rcvCnt=2060037, sentCnt=2060041, reserved=true, lastAck=2060032,
nodeLeft=false, node=TcpDiscoveryNode
[id=62c088cc-f6a8-49b2-8eee-911f147d99b2, addrs=[10.166.161.111, 127.0.0.1],
sockAddrs=[/127.0.0.1:0, gi2p1xrcds001.gi02.bpty/10.166.161.111:0],
discPort=0, order=109, intOrder=61, lastExchangeTime=1539342673260,
loc=false, ver=1.8.0#20161205-sha1:9ca40dbe, isClient=true], connected=true,
connectCnt=0, queueLimit=100000, reserveCnt=68], super=GridNioSessionImpl
[locAddr=/10.166.186.158:9090, rmtAddr=/10.166.161.111:45006,
createTime=1539921738152, closeTime=0, bytesSent=6593820,
bytesRcvd=20615839, sndSchedTime=1539929665103, lastSndTime=1539921785427,
lastRcvTime=1539929665103, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@40e27086, directMode=true],
GridConnectionBytesVerifyFilter, SSL filter], accepted=true]]
[02:15:14,320][WARN
][grid-nio-worker-1-#10%CasinoApacheIgniteServices%][TcpCommunicationSpi]
Closing NIO session because of unhandled exception [cls=class
o.a.i.i.util.nio.GridNioException, msg=Connection timed out]
When we timed out the hung thread, we can see the below stack trace in
client service.
02:08:40,697 - ERROR - pool-45-thread-1 - CASINO_DATASYNC_LOGGER Exception
in Syncing the Data for LMTEMPLATECONFIG
javax.cache.CacheException: class
org.apache.ignite.IgniteInterruptedException: null
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1440)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:2183)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.putAll(IgniteCacheProxy.java:1430)
at
com.pg.casino.service.datasync.DataSyncManager.syncDataIntoIgnite(DataSyncManager.java:835)
at
com.pg.casino.service.datasync.LMDataSyncManager.syncGenericLobbyTemplateConfig(LMDataSyncManager.java:327)
at
com.pg.casino.service.datasync.DataSyncCallable.call(DataSyncCallable.java:118)
at
com.pg.casino.service.datasync.DataSyncCallable.call(DataSyncCallable.java:22)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: class org.apache.ignite.IgniteInterruptedException: null
at
org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:766)
at
org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:764)
... 11 more
Caused by: java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:996)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:160)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:118)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.putAll(GridDhtAtomicCache.java:704)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.putAll(IgniteCacheProxy.java:1423)
*cache info in Ignite xml :*
<bean id="lMTemplateIdVsJsonCacheIgnite"
class="org.apache.ignite.configuration.CacheConfiguration"
scope="singleton">
<property name="name" value="lMTemplateIdVsJsonCacheIgnite"/>
<property name="cacheMode" value="REPLICATED"/>
<property name="memoryMode" value="ONHEAP_TIERED"/>
<property name="offHeapMaxMemory" value="#{1 * 20L * 1024L * 1024L}"/>
<property name="backups" value="1"/>
<property name="writeSynchronizationMode" value="FULL_SYNC"/>
<property name="startSize" value="#{5 * 1024 * 1024}"/>
<property name="swapEnabled" value="false"/>
<property name="rebalanceMode" value="ASYNC"/>
<property name="rebalanceBatchSize" value="#{1024 * 1024}"/>
<property name="rebalanceThrottle" value="0"/>
<property name="rebalanceThreadPoolSize" value="4"/>
</bean>
*
Ignite config:*
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean id="ignite.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="gridName" value="CasinoApacheIgniteServices"/>
<property name="peerClassLoadingEnabled" value="false"/>
<property name="metricsUpdateFrequency" value="2000" />
<property name="discoverySpi">
<bean
class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<value>#$IGNITE_CASINO_SERVICE_1_IP$#:49500</value>
<value>#$IGNITE_CASINO_SERVICE_2_IP$#:49500</value>
<value>#$IGNITE_CASINO_SERVICE_3_IP$#:49500</value>
</list>
</property>
</bean>
</property>
<property name="localPort" value="49500"/>
<property name="heartbeatFrequency" value="2000"/>
<property name="maxMissedHeartbeats" value="5"/>
<property name="reconnectCount" value="100"/>
<property name="networkTimeout" value="10000"/>
<property name="socketTimeout" value="10000"/>
<property name="ackTimeout" value="10000"/>
<property name="statisticsPrintFrequency" value="60000"/>
</bean>
</property>
<property name="communicationSpi">
<bean
class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
<property name="slowClientQueueLimit" value="18000"/>
<property name="messageQueueLimit" value="20000"/>
<property name="localPort" value="9090"/>
<property name="connectTimeout" value="5000"/>
<property name="idleConnectionTimeout" value="75000"/>
<property name="sharedMemoryPort" value="-1"/>
</bean>
</property>
</bean>
</beans>
Thanks,
Praveen
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Inserting data into Ignite thread is hung
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Nothing obviously wrong (or happening at all) in your thread dumps.
Unfortunately I don't think the community will help you with 1.8 much,
since it's a very old release as far we are concerned.
Regards,
--
Ilya Kasnacheev
пт, 26 окт. 2018 г. в 9:39, praveeng <pr...@gmail.com>:
> Hi,
>
> We are getting this issue in production continuously.
> I am attaching the thread dumps of my services (ignite client
> -threaddump14168.txt file and ignite servers- 3 files ).
> can any one please analyze and suggest once how to fix this issue.
>
> threaddump14168.txt
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t1753/threaddump14168.txt>
>
> CAI.7z <http://apache-ignite-users.70518.x6.nabble.com/file/t1753/CAI.7z>
>
>
> Thanks,
> Praveen
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Inserting data into Ignite thread is hung
Posted by praveeng <pr...@gmail.com>.
Hi,
We are getting this issue in production continuously.
I am attaching the thread dumps of my services (ignite client
-threaddump14168.txt file and ignite servers- 3 files ).
can any one please analyze and suggest once how to fix this issue.
threaddump14168.txt
<http://apache-ignite-users.70518.x6.nabble.com/file/t1753/threaddump14168.txt>
CAI.7z <http://apache-ignite-users.70518.x6.nabble.com/file/t1753/CAI.7z>
Thanks,
Praveen
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Inserting data into Ignite thread is hung
Posted by praveeng <pr...@gmail.com>.
Hi,
I don't think there is a network issue, since at that moment other clients
are able to access the data from the Ignite cluster. But some times the
inserting thread is hung for 15 mins, after that it has proceeded further.
If the thread is hung which is inserting, we are canceling the thread
explicitly by the below code.
That's why you can see IgniteInterruptedException in the logs.
But why the thread is becoming hung while inserting the data[ max 10
records].
if(futureReturn.size() > 0){
for(Future<Boolean> future : futureReturn){
try{
Boolean result = future.get(20, TimeUnit.MINUTES);
if(!result){
dataSyncSucces = false;
break;
}
}catch(Exception e){
future.cancel(true);
dataSyncSucces = false;
}
}
}
Even the same issue i observed in local services also but at different piece
of code.
Thanks,
Praveen
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Inserting data into Ignite thread is hung
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
This might be due genuine network problems between two nodes, or maybe it
is something like https://issues.apache.org/jira/browse/IGNITE-8153 - a
very rare SSL bug which will only manifest under specific conditions.
Regards,
--
Ilya Kasnacheev
пн, 22 окт. 2018 г. в 13:53, praveeng <pr...@gmail.com>:
> Hi ,
>
> I have forgot to specify the below points.
>
> 1. This issue is happening every time at the same place of code for the
> same
> cache.
> 2. The cache to which we are inserting is not at all used by any other
> service [ cache instance is created in other services but we will not be
> making any call to this cache. At present we are just inserting the data
> into the cache by client1 and other clients just creating the cache
> instance
> to this cache but not making any call.]
>
>
> Thanks,
> Praveen
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Inserting data into Ignite thread is hung
Posted by praveeng <pr...@gmail.com>.
Hi ,
I have forgot to specify the below points.
1. This issue is happening every time at the same place of code for the same
cache.
2. The cache to which we are inserting is not at all used by any other
service [ cache instance is created in other services but we will not be
making any call to this cache. At present we are just inserting the data
into the cache by client1 and other clients just creating the cache instance
to this cache but not making any call.]
Thanks,
Praveen
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/