You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by "Gohil, Shailendrasinh (INTL)" <Sh...@salientcrgt.com> on 2018/07/12 15:04:02 UTC
Possible starvation in striped pool
We are working on integrating the JEE web application with Apache Ignite and caching around 900 millions records from Oracle database to off heap cache. The client was started from WebLogic using the servlet listener. When users are trying to access the application, the application queries their data from cache. We see the below issue when there are more than 2 users performing the similar operation on their own data. This was not the performance we expected from the documentation.
WARN [org.apache.ignite.internal.util.typedef.G] - >>> Possible starvation in striped pool.
Thread name: sys-stripe-14-#15%AppCluster%
Queue: []
Deadlock: false
Completed: 4
Thread [name="sys-stripe-14-#15% AppCluster%", id=107, state=WAITING, blockCnt=0, waitCnt=6]
Lock [object=java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@5a7be62, ownerName=exchange-worker-#57% AppCluster%, ownerId=171]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:317)
at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
at java.lang.Thread.run(Thread.java:748)
Re: Possible starvation in striped pool
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
At this point I recommend debugging which statements are ran on Oracle and
why they take long.
Also I have noticed: appDataSource - is it behind some kind of connection
pool? I am afraid it is possible that this data source is single-threaded
in the absense of connection pool, hence you might see contention.
Regards,
--
Ilya Kasnacheev
2018-07-18 20:48 GMT+03:00 Shailendrasinh Gohil <
Shailendrasinh.Gohil@salientcrgt.com>:
> Here you go...
>
> <bean class="org.apache.ignite.configuration.CacheConfiguration">
> <property name="name" value="GeneratedProdIdCache" />
> <property name="cacheMode" value="PARTITIONED" />
> <property name="atomicityMode" value="TRANSACTIONAL" />
> <property name="partitionLossPolicy" value="READ_WRITE_SAFE"/>
>
> <property name="cacheStoreFactory">
> <bean
> class="org.apache.ignite.cache.store.jdbc.
> CacheJdbcPojoStoreFactory">
> <property name="dataSourceBean"
> value="appDataSource" />
> <property name="dialect">
> <bean class="org.apache.ignite.
> cache.store.jdbc.dialect.OracleDialect">
> </bean>
> </property>
>
> <property name="types">
> <list>
> <bean class="org.apache.ignite.
> cache.store.jdbc.JdbcType">
> <property name="cacheName"
> value="GeneratedProdIdCache" />
> <property name="keyType"
> value="java.lang.String" />
> <property name="valueType"
>
> value="ignite.model.GeneratedProdId" />
> <property
> name="databaseSchema" value="APPSCHEMA" />
> <property
> name="databaseTable" value="GENERATED_PROD_ID" />
>
> <property name="keyFields">
> <list>
> <bean
> class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
>
> <constructor-arg>
>
> <util:constant static-field="java.sql.Types.VARCHAR" />
>
> </constructor-arg>
>
> <constructor-arg value="PROD_ID" />
>
> <constructor-arg value="java.lang.String" />
>
> <constructor-arg value="prodId" />
> </bean>
> </list>
> </property>
>
> <property
> name="valueFields">
> <list>
> <bean
> class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
>
> <constructor-arg>
>
> <util:constant static-field="java.sql.Types.VARCHAR" />
>
> </constructor-arg>
>
> <constructor-arg value="PROD_ID" />
>
> <constructor-arg value="java.lang.String" />
>
> <constructor-arg value="prodId" />
> </bean>
>
> <bean
> class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
>
> <constructor-arg>
>
> <util:constant static-field="java.sql.Types.VARCHAR" />
>
> </constructor-arg>
>
> <constructor-arg value="PROD_ID_KEY" />
>
> <constructor-arg value="java.lang.String" />
>
> <constructor-arg value="prodIdKey" />
> </bean>
>
> </list>
> </property>
> </bean>
> </list>
> </property>
> </bean>
> </property>
>
> <property name="readThrough" value="true" />
> <property name="writeThrough" value="true" />
> <property name="writeBehindEnabled" value="true" />
> <property name="writeBehindFlushSize" value="2048" />
> <property name="writeBehindFlushFrequency" value="0" />
> <property name="writeBehindFlushThreadCount" value="1" />
> <property name="writeBehindBatchSize" value="512" />
>
> <property name="queryEntities">
> <list>
> <bean class="org.apache.ignite.cache.QueryEntity">
> <property name="keyType"
> value="java.lang.String" />
> <property name="valueType"
> value="ignite.model.GeneratedProdId"
> />
> <property name="tableName"
> value="GENERATED_PROD_ID" />
> <property name="keyFieldName"
> value="prodId" />
>
> <property name="keyFields">
> <list>
> <value>prodId</value>
> </list>
> </property>
>
> <property name="fields">
> <map>
> <entry key="prodId"
> value="java.lang.String" />
> <entry key="prodIdKey"
> value="java.lang.String" />
> </map>
> </property>
>
> <property name="aliases">
> <map>
> <entry key="prodId"
> value="PROD_ID" />
> <entry key="prodIdKey"
> value="PROD_ID_KEY" />
> </map>
> </property>
>
> <property name="indexes">
> <list>
> <bean
> class="org.apache.ignite.cache.QueryIndex">
> <property
> name="name" value="INDX_GNRTD_PROD_ID" />
> <property
> name="indexType" value="FULLTEXT" />
>
> <property
> name="fields">
> <map>
>
> <entry key="prodId" value="true" />
> </map>
> </property>
> </bean>
> </list>
> </property>
> </bean>
> </list>
> </property>
> </bean>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Possible starvation in striped pool
Posted by Shailendrasinh Gohil <Sh...@salientcrgt.com>.
Here you go...
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="GeneratedProdIdCache" />
<property name="cacheMode" value="PARTITIONED" />
<property name="atomicityMode" value="TRANSACTIONAL" />
<property name="partitionLossPolicy" value="READ_WRITE_SAFE"/>
<property name="cacheStoreFactory">
<bean
class="org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory">
<property name="dataSourceBean" value="appDataSource" />
<property name="dialect">
<bean class="org.apache.ignite.cache.store.jdbc.dialect.OracleDialect">
</bean>
</property>
<property name="types">
<list>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcType">
<property name="cacheName" value="GeneratedProdIdCache" />
<property name="keyType" value="java.lang.String" />
<property name="valueType"
value="ignite.model.GeneratedProdId" />
<property name="databaseSchema" value="APPSCHEMA" />
<property name="databaseTable" value="GENERATED_PROD_ID" />
<property name="keyFields">
<list>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg>
<util:constant static-field="java.sql.Types.VARCHAR" />
</constructor-arg>
<constructor-arg value="PROD_ID" />
<constructor-arg value="java.lang.String" />
<constructor-arg value="prodId" />
</bean>
</list>
</property>
<property name="valueFields">
<list>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg>
<util:constant static-field="java.sql.Types.VARCHAR" />
</constructor-arg>
<constructor-arg value="PROD_ID" />
<constructor-arg value="java.lang.String" />
<constructor-arg value="prodId" />
</bean>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg>
<util:constant static-field="java.sql.Types.VARCHAR" />
</constructor-arg>
<constructor-arg value="PROD_ID_KEY" />
<constructor-arg value="java.lang.String" />
<constructor-arg value="prodIdKey" />
</bean>
</list>
</property>
</bean>
</list>
</property>
</bean>
</property>
<property name="readThrough" value="true" />
<property name="writeThrough" value="true" />
<property name="writeBehindEnabled" value="true" />
<property name="writeBehindFlushSize" value="2048" />
<property name="writeBehindFlushFrequency" value="0" />
<property name="writeBehindFlushThreadCount" value="1" />
<property name="writeBehindBatchSize" value="512" />
<property name="queryEntities">
<list>
<bean class="org.apache.ignite.cache.QueryEntity">
<property name="keyType" value="java.lang.String" />
<property name="valueType"
value="ignite.model.GeneratedProdId" />
<property name="tableName" value="GENERATED_PROD_ID" />
<property name="keyFieldName" value="prodId" />
<property name="keyFields">
<list>
<value>prodId</value>
</list>
</property>
<property name="fields">
<map>
<entry key="prodId" value="java.lang.String" />
<entry key="prodIdKey" value="java.lang.String" />
</map>
</property>
<property name="aliases">
<map>
<entry key="prodId" value="PROD_ID" />
<entry key="prodIdKey" value="PROD_ID_KEY" />
</map>
</property>
<property name="indexes">
<list>
<bean class="org.apache.ignite.cache.QueryIndex">
<property name="name" value="INDX_GNRTD_PROD_ID" />
<property name="indexType" value="FULLTEXT" />
<property name="fields">
<map>
<entry key="prodId" value="true" />
</map>
</property>
</bean>
</list>
</property>
</bean>
</list>
</property>
</bean>
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Possible starvation in striped pool
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello again!
I have just noticed the following stack trace:
"flusher-0-#588%AppCluster%" #633 prio=5 os_prio=0
tid=0x00007f18d424f800 nid=0xe1bb runnable [0x00007f197c1cd000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at oracle.net.ns.Packet.receive(Packet.java:311)
at oracle.net.ns.DataPacket.receive(DataPacket.java:105)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:305)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:249)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:171)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:89)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:123)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:79)
at oracle.jdbc.driver.T4CMAREngineStream.unmarshalUB1(T4CMAREngineStream.java:429)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:397)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:587)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:225)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:53)
at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:943)
at oracle.jdbc.driver.OraclePreparedStatement.executeForRowsWithTimeout(OraclePreparedStatement.java:12029)
at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:12140)
- locked <0x00007f2aaa591778> (a oracle.jdbc.driver.T4CConnection)
at oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:246)
at org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore.executeBatch(CacheAbstractJdbcStore.java:1226)
at org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore.writeAll(CacheAbstractJdbcStore.java:1111)
at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.updateStore(GridCacheWriteBehindStore.java:809)
at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.applyBatch(GridCacheWriteBehindStore.java:725)
at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.access$2400(GridCacheWriteBehindStore.java:75)
at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore$Flusher.flushCacheCoalescing(GridCacheWriteBehindStore.java:1113)
at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore$Flusher.body(GridCacheWriteBehindStore.java:1011)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:748)
It looks like you're waiting for Oracle to complete batch execution, but it
won't.
Regards,
--
Ilya Kasnacheev
2018-07-18 17:47 GMT+03:00 Ilya Kasnacheev <il...@gmail.com>:
> Hello!
>
> Can you please share the configuration of your Apache Ignite nodes,
> especially the cache store's of caches. I have just noticed that you're
> actually waiting on cache store lock.
>
> Regards,
>
> --
> Ilya Kasnacheev
>
> 2018-07-17 19:11 GMT+03:00 Shailendrasinh Gohil <Shailendrasinh.Gohil@
> salientcrgt.com>:
>
>> We are using the TreeMap for all the putAll operations. We also tried
>> streamer API to create the automatic batches. Still the issue is same.
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
>
Re: Possible starvation in striped pool
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Can you please share the configuration of your Apache Ignite nodes,
especially the cache store's of caches. I have just noticed that you're
actually waiting on cache store lock.
Regards,
--
Ilya Kasnacheev
2018-07-17 19:11 GMT+03:00 Shailendrasinh Gohil <
Shailendrasinh.Gohil@salientcrgt.com>:
> We are using the TreeMap for all the putAll operations. We also tried
> streamer API to create the automatic batches. Still the issue is same.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Possible starvation in striped pool
Posted by Shailendrasinh Gohil <Sh...@salientcrgt.com>.
We are using the TreeMap for all the putAll operations. We also tried
streamer API to create the automatic batches. Still the issue is same.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Possible starvation in striped pool
Posted by Sambhaji Sawant <sa...@gmail.com>.
Hello same issue occurred when trying to put object in cache using
cache.put method.after changing put to putAsync issue was solved.
I have read about when you using putAll methode pass sorted collection to
it so it avoid deadlock. So is it true?
On Tue, Jul 17, 2018, 8:22 PM ilya.kasnacheev <il...@gmail.com>
wrote:
> Hello!
>
> I have noticed that you are using putAll in your code.
>
> Apache Ignite is susceptible to deadlocks in the same fashion as regular
> multi-threaded code: i.e., if you take multiple locks (as putAll does, on
> partitions for its keys), you can get deadlock unless you maintain sequence
> of locks, i.e., always lock B after A and not the other way around.
>
> You can attain that by passing TreeMap's to putAll as opposed to HashMap's.
>
> Can you try to pass TreeMap's to putAll, see if it fixed your issue?
>
> Regards,
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Possible starvation in striped pool
Posted by "ilya.kasnacheev" <il...@gmail.com>.
Hello!
I have noticed that you are using putAll in your code.
Apache Ignite is susceptible to deadlocks in the same fashion as regular
multi-threaded code: i.e., if you take multiple locks (as putAll does, on
partitions for its keys), you can get deadlock unless you maintain sequence
of locks, i.e., always lock B after A and not the other way around.
You can attain that by passing TreeMap's to putAll as opposed to HashMap's.
Can you try to pass TreeMap's to putAll, see if it fixed your issue?
Regards,
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Possible starvation in striped pool
Posted by Shailendrasinh Gohil <Sh...@salientcrgt.com>.
Please find attached thread dump as requested.
ServerThreadDump0716.txt
<http://apache-ignite-users.70518.x6.nabble.com/file/t1917/ServerThreadDump0716.txt>
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Possible starvation in striped pool
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Can you please provide the thread dump of problematic cluster after removal
of close statements on caches?
Regards,
--
Ilya Kasnacheev
2018-07-16 17:21 GMT+03:00 Shailendrasinh Gohil <
Shailendrasinh.Gohil@salientcrgt.com>:
> Thanks again for the response.
>
> We have tried removing the close statements but the result was same. And
> yes, other threads accessing cache from the same Dao.
>
> We also tried both the atomicityMode to see if any improvement. We also
> have
> write behind enabled for the large tables with frequent get and put
> operations.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Possible starvation in striped pool
Posted by Shailendrasinh Gohil <Sh...@salientcrgt.com>.
Thanks again for the response.
We have tried removing the close statements but the result was same. And
yes, other threads accessing cache from the same Dao.
We also tried both the atomicityMode to see if any improvement. We also have
write behind enabled for the large tables with frequent get and put
operations.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Possible starvation in striped pool
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
I can see here that you are trying to destroy a cache:
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
at
org.apache.ignite.internal.util.future.IgniteFutureImpl.get(IgniteFutureImpl.java:134)
at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.close(GatewayProtectedCacheProxy.java:1514)
at
dao.impl.IgniteLookupServiceImpl.getProdCountForEst(IgniteLookupServiceImpl.java:2645)
at
service.impl.RegistrationServiceImpl.getProdCountForEst(RegistrationServiceImpl.java:2588)
Is it supposed to be destroyed in this method? I can see other threads
accessing cache from the same Dao.
Regards,
--
Ilya Kasnacheev
2018-07-12 19:35 GMT+03:00 Shailendrasinh Gohil <
Shailendrasinh.Gohil@salientcrgt.com>:
> Thank you for your response. Please find attached thread dumps for client
> and
> server nodes. ClientThreadDump.txt
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> t1917/ClientThreadDump.txt>
> ThreadDumpServer1.txt
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> t1917/ThreadDumpServer1.txt>
> ThreadDumpServer2.txt
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> t1917/ThreadDumpServer2.txt>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Possible starvation in striped pool
Posted by Shailendrasinh Gohil <Sh...@salientcrgt.com>.
Thank you for your response. Please find attached thread dumps for client and
server nodes. ClientThreadDump.txt
<http://apache-ignite-users.70518.x6.nabble.com/file/t1917/ClientThreadDump.txt>
ThreadDumpServer1.txt
<http://apache-ignite-users.70518.x6.nabble.com/file/t1917/ThreadDumpServer1.txt>
ThreadDumpServer2.txt
<http://apache-ignite-users.70518.x6.nabble.com/file/t1917/ThreadDumpServer2.txt>
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Possible starvation in striped pool
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Unfortunately it's hard to say without looking at full thread dumps from
all nodes in cluster. Can you post them somewhere?
Regards,
--
Ilya Kasnacheev
2018-07-12 18:04 GMT+03:00 Gohil, Shailendrasinh (INTL) <
Shailendrasinh.Gohil@salientcrgt.com>:
> We are working on integrating the JEE web application with Apache Ignite
> and caching around 900 millions records from Oracle database to off heap
> cache. The client was started from WebLogic using the servlet listener.
> When users are trying to access the application, the application queries
> their data from cache. We see the below issue when there are more than 2
> users performing the similar operation on their own data. This was not the
> performance we expected from the documentation.
>
>
>
> WARN [org.apache.ignite.internal.util.typedef.G] - >>> Possible
> starvation in striped pool.
>
> Thread name: sys-stripe-14-#15%AppCluster%
>
> Queue: []
>
> Deadlock: false
>
> Completed: 4
>
> Thread [name="sys-stripe-14-#15% AppCluster%", id=107, state=WAITING,
> blockCnt=0, waitCnt=6]
>
> Lock [object=java.util.concurrent.locks.ReentrantReadWriteLock$
> NonfairSync@5a7be62, ownerName=exchange-worker-#57% AppCluster%,
> ownerId=171]
>
> at sun.misc.Unsafe.park(Native Method)
>
> at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.
> parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.
> doAcquireShared(AbstractQueuedSynchronizer.java:967)
>
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.
> acquireShared(AbstractQueuedSynchronizer.java:1283)
>
> at java.util.concurrent.locks.ReentrantReadWriteLock$
> ReadLock.lock(ReentrantReadWriteLock.java:727)
>
> at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(
> GridCacheIoManager.java:317)
>
> at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(
> GridCacheIoManager.java:304)
>
> at o.a.i.i.processors.cache.GridCacheIoManager.access$100(
> GridCacheIoManager.java:99)
>
> at o.a.i.i.processors.cache.GridCacheIoManager$1.
> onMessage(GridCacheIoManager.java:293)
>
> at o.a.i.i.managers.communication.GridIoManager.
> invokeListener(GridIoManager.java:1555)
>
> at o.a.i.i.managers.communication.GridIoManager.
> processRegularMessage0(GridIoManager.java:1183)
>
> at o.a.i.i.managers.communication.GridIoManager.
> access$4200(GridIoManager.java:126)
>
> at o.a.i.i.managers.communication.GridIoManager$9.
> run(GridIoManager.java:1090)
>
> at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.
> java:505)
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
>
>