[jira] [Created] (KARAF-4878) Cellar Hazelcast unresponsive when ETH Down

["Suresh Perumal (JIRA)" <ji...@apache.org>]

Suresh Perumal created KARAF-4878:
-------------------------------------

             Summary: Cellar Hazelcast unresponsive when ETH Down
                 Key: KARAF-4878
                 URL: https://issues.apache.org/jira/browse/KARAF-4878
             Project: Karaf
          Issue Type: Bug
          Components: cellar-hazelcast
    Affects Versions: 4.0.5
         Environment: Redhat Linux 7.2, CentOS 7.2
            Reporter: Suresh Perumal
            Priority: Blocker


Cluster is configured with 2 Nodes. They are up and running.

As part of fail-over scenario simulation. We are trying to test "ETHERNET down scenario" by running "/etc/sysconfig/network-scripts/ifdown eth0" command on the first node.

During this scenario we are shutting down the first node where the ETH is  down by using monitoring scripts(in-house scripts). The second node(Among those two nodes) is kept alive.

Second Node's Hazelcast is not accessible for more than 15 minutes. We are getting bellow exception and no operation related to Hazelcast is working. Applications whichever uses hazelcast kept frozen.

Invocation                       | 52 - com.hazelcast - 3.5.2 | [10.249.50.80]:5701 [cellar] [3.5.2] While asking 'is-executing': Invocation{ serviceName='hz:impl:mapService', op=PutOperation{unacknowledged-alarm}, partitionId=165, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[10.249.50.79]:5701, backupsExpected=0, backupsCompleted=0}
java.util.concurrent.TimeoutException: Call Invocation{ serviceName='hz:impl:mapService', op=com.hazelcast.spi.impl.operationservice.impl.operations.IsStillExecutingOperation{serviceName='hz:impl:mapService', partitionId=-1, callId=2114, invocationTime=1480511190143, waitTimeout=-1, callTimeout=5000}, partitionId=-1, replicaIndex=0, tryCount=0, tryPauseMillis=0, invokeCount=1, callTimeout=5000, target=Address[10.249.50.79]:5701, backupsExpected=0, backupsCompleted=0} encountered a timeout
        at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:366)[52:com.hazelcast:3.5.2]
        at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:334)[52:com.hazelcast:3.5.2]
        at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225)[52:com.hazelcast:3.5.2]
        at com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService.isOperationExecuting(IsStillRunningService.java:85)[52:com.hazelcast:3.5.2]
        at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.waitForResponse(InvocationFuture.java:275)[52:com.hazelcast:3.5.2]
        at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:224)[52:com.hazelcast:3.5.2]
        at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204)[52:com.hazelcast:3.5.2]
        at com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:456)[52:com.hazelcast:3.5.2]
        at com.hazelcast.map.impl.proxy.MapProxySupport.putInternal(MapProxySupport.java:417)[52:com.hazelcast:3.5.2]
        at com.hazelcast.map.impl.proxy.MapProxyImpl.put(MapProxyImpl.java:97)[52:com.hazelcast:3.5.2]
        at com.hazelcast.map.impl.proxy.MapProxyImpl.put(MapProxyImpl.java:87)[52:com.hazelcast:3.5.2]
        at com.fujitsu.fnc.emf.fpmplatform.cachemanager.HazelcastCacheManagerMapServiceImpl.addToMap(HazelcastCacheManagerMapServiceImpl.java:87)[209:FPMHazelcastCache:4.1.0.SNAPSHOT]
        at Proxy1897a82c_c032_4a5c_9839_e71cb2af452a.addToMap(Unknown Source)[:]
        at com.fujitsu.fnc.ngemf.fm.server.impl.FpmConsumerTask.prepareJSON(FpmConsumerTask.java:151)[235:com.fujitsu.fnc.ngemf.fm.server.impl:4.1.0.SNAPSHOT]
        at com.fujitsu.fnc.ngemf.fm.server.impl.FpmConsumerTask.run(FpmConsumerTask.java:244)[235:com.fujitsu.fnc.ngemf.fm.server.impl:4.1.0.SNAPSHOT]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_66]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_66]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_66]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_66]
        at java.lang.Thread.run(Thread.java:745)[:1.8.0_66]




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)