You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Enrico Olivelli (JIRA)" <ji...@apache.org> on 2015/09/22 16:21:04 UTC
[jira] [Comment Edited] (IGNITE-1524) Client node throws exception
when server node killed
[ https://issues.apache.org/jira/browse/IGNITE-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14902679#comment-14902679 ]
Enrico Olivelli edited comment on IGNITE-1524 at 9/22/15 2:20 PM:
------------------------------------------------------------------
I think I got into a similar issue
even with forceservermode=true the client cannot ever recover from the shutdown of all the "server" nodes, while accessing a partioned cache
using 1.3.0-incubating
should I file another issue ?
{code}
org.apache.ignite.cache.CacheServerNotFoundException: Failed to map keys for cache (all partition nodes left the grid).
at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1571)
at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:1689)
at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(IgniteCacheProxy.java:963)
at magnews.cache.IgniteCacheFactory$IgniteCacheFacade.put(IgniteCacheFactory.java:392)
at magnews.cache.IgniteClientSviluppoRef3Example.go(IgniteClientSviluppoRef3Example.java:50)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Caused by: class org.apache.ignite.internal.cluster.ClusterTopologyServerNotFoundException: Failed to map keys for cache (all partition nodes left the grid).
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.map0(GridNearAtomicUpdateFuture.java:664)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.mapOnTopology(GridNearAtomicUpdateFuture.java:616)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.map(GridNearAtomicUpdateFuture.java:327)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$14.apply(GridDhtAtomicCache.java:776)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$14.apply(GridDhtAtomicCache.java:774)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:580)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsync0(GridDhtAtomicCache.java:774)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:317)
at org.apache.ignite.internal.processors.cache.GridCacheAdapter.putAsync(GridCacheAdapter.java:2292)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:293)
at org.apache.ignite.internal.processors.cache.distributed.near.GridNearAtomicCache.put(GridNearAtomicCache.java:400)
at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:1997)
at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(IgniteCacheProxy.java:956)
... 25 more
{code}
And after some time I get
{code}
java.lang.IllegalStateException: Grid is in invalid state to perform this operation. It either not started yet or has already being or have stopped [gridName=SVILUPPOREF]
{code}
was (Author: eolivelli):
I think I got into a similar issue
even with forceservermode=true the client cannot ever recover from the shutdown of all the "server" nodes, while accessing a partioned cache
using 1.3.0-incubating
should I file another issue ?
{code}
org.apache.ignite.cache.CacheServerNotFoundException: Failed to map keys for cache (all partition nodes left the grid).
at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1571)
at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:1689)
at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(IgniteCacheProxy.java:963)
at magnews.cache.IgniteCacheFactory$IgniteCacheFacade.put(IgniteCacheFactory.java:392)
at magnews.cache.IgniteClientSviluppoRef3Example.go(IgniteClientSviluppoRef3Example.java:50)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Caused by: class org.apache.ignite.internal.cluster.ClusterTopologyServerNotFoundException: Failed to map keys for cache (all partition nodes left the grid).
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.map0(GridNearAtomicUpdateFuture.java:664)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.mapOnTopology(GridNearAtomicUpdateFuture.java:616)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.map(GridNearAtomicUpdateFuture.java:327)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$14.apply(GridDhtAtomicCache.java:776)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$14.apply(GridDhtAtomicCache.java:774)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:580)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsync0(GridDhtAtomicCache.java:774)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:317)
at org.apache.ignite.internal.processors.cache.GridCacheAdapter.putAsync(GridCacheAdapter.java:2292)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:293)
at org.apache.ignite.internal.processors.cache.distributed.near.GridNearAtomicCache.put(GridNearAtomicCache.java:400)
at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:1997)
at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(IgniteCacheProxy.java:956)
... 25 more
{code}
> Client node throws exception when server node killed
> ----------------------------------------------------
>
> Key: IGNITE-1524
> URL: https://issues.apache.org/jira/browse/IGNITE-1524
> Project: Ignite
> Issue Type: Bug
> Reporter: Artem Shutak
> Assignee: Semen Boikov
> Fix For: ignite-1.4
>
> Attachments: logs-20150922-120712.zip
>
>
> In process of working on IGNITE-1397 I've found that Ignite client node throws CacheException which caused by IgniteClientDisconnectedException (stack trace below) sometimes when I kill one server node at cluster.
> My configuration:
> - 3 server nodes
> - 2 client nodes
> - discovery configuration:
> {code}
> <property name="discoverySpi">
> <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> <property name="ipFinder">
> <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> <property name="addresses">
> <list>
> <value>127.0.0.1:47500</value>
> <value>127.0.0.1:47501</value>
> <value>127.0.0.1:47502</value>
> </list>
> </property>
> </bean>
> </property>
> </bean>
> </property>
> {code}
> Test description:
> - client nodes make random atomic ops to cluster (pts, gets, invoke, remove)
> - the first client node has a thread which kill with -9 and start one server one by one. Kills every 60 seconds and run after 10 seconds.
> Client node fails not always.
> {noformat}
> javax.cache.CacheException: class org.apache.ignite.IgniteClientDisconnectedException: Operation has been cancelled (client node disconnected).
> at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1621)
> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:1749)
> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.get(IgniteCacheProxy.java:815)
> at org.apache.ignite.yardstick.cache.failover.IgniteAtomicRetriesFailoverBenchmark.test(IgniteAtomicRetriesFailoverBenchmark.java:39)
> at org.yardstickframework.impl.BenchmarkRunner$2.run(BenchmarkRunner.java:165)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: class org.apache.ignite.IgniteClientDisconnectedException: Operation has been cancelled (client node disconnected).
> at org.apache.ignite.internal.util.IgniteUtils$14.apply(IgniteUtils.java:801)
> at org.apache.ignite.internal.util.IgniteUtils$14.apply(IgniteUtils.java:799)
> ... 6 more
> Caused by: class org.apache.ignite.internal.IgniteClientDisconnectedCheckedException: Operation has been cancelled (client node disconnected).
> at org.apache.ignite.internal.processors.cache.GridCacheMvccManager.disconnectedError(GridCacheMvccManager.java:360)
> at org.apache.ignite.internal.processors.cache.GridCacheMvccManager.onDisconnected(GridCacheMvccManager.java:334)
> at org.apache.ignite.internal.processors.cache.GridCacheSharedContext.onDisconnected(GridCacheSharedContext.java:141)
> at org.apache.ignite.internal.processors.cache.GridCacheProcessor.onDisconnected(GridCacheProcessor.java:997)
> at org.apache.ignite.internal.IgniteKernal.onDisconnected(IgniteKernal.java:2983)
> at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$5.onDiscovery(GridDiscoveryManager.java:592)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.notifyDiscovery(ClientImpl.java:2024)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.notifyDiscovery(ClientImpl.java:2005)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1422)
> at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)