Too many open cursors issue after cache partition loss exceptions

[Sumit Deshinge <su...@gmail.com>]

Hi,

I am adding a large amount of data into ignite cache using multiple thin
clients hosted on different machines. Ignite cache configuration is
FULL_SYNC with REPLICATED mode.

After few hours of data addition, I can see multiple below exceptions:
org.apache.ignite.internal.processors.cache.CacheInvalidStateException:
Failed to execute query because cache partition has been lostPart.

And after observing multiple above exceptions, ignite started throwing 'too
many open cursors issue'.
org.apache.ignite.internal.processors.platform.client.IgniteClientException:
Too many open cursors (either close other open cursors or increase the
limit through ClientConnectorConfiguration.maxOpenCursorsPerConnection)
[maximum=128, current=128]
at
org.apache.ignite.internal.processors.platform.client.ClientConnectionContext.incrementCursors(ClientConnectionContext.java:264)
at o........

This open cursors leaks seems to be through ignite internal workflows,
because all the query cursor usages in my code is made sure to close in
finally block. And if there is no cache partition loss error, then too many
open cursors issue is also not observed.

So there are 2 queries:
1. What would cause cache partition loss in 'FULL_SYNC' and 'REPLICATED'
mode ?
2. Why there is 'Too many open cursors' issue after
'CacheInvalidStateException: Failed to execute query because cache
partition has been lostPart' issue is observed. After this, cluster becomes
not usable. What is the way to avoid or recover from this situation ?



-- 
Regards,
Sumit Deshinge