[jira] [Created] (CLOUDSTACK-7651) Various operation intermittently fail in VMware with 'he object has already been deleted or has not been completely created' exceptions

["Likitha Shetty (JIRA)" <ji...@apache.org>]

Likitha Shetty created CLOUDSTACK-7651:
------------------------------------------

             Summary: Various operation intermittently fail in VMware with 'he object has already been deleted or has not been completely created' exceptions
                 Key: CLOUDSTACK-7651
                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7651
             Project: CloudStack
          Issue Type: Bug
      Security Level: Public (Anyone can view this level - this is the default.)
          Components: VMware
    Affects Versions: 4.5.0
            Reporter: Likitha Shetty
            Assignee: Likitha Shetty
            Priority: Critical
             Fix For: 4.5.0


+Steps to reproduce+ 
1. Setup VMware 4.3.0.1 with 1 CCP Management server.
2. Create 2 VMware DCs in 2 different vCenters.
3. Add 2 zones in CCP corresponding to the two DCs.
4. Do basic VM/Volume lifecycle operations.

Keep monitoring the logs for the following error - *javax.xml.ws.soap.SOAPFaultException: The object has already been deleted or has not been completely created*

For e.g. 
{noformat}
javax.xml.ws.soap.SOAPFaultException
Message: The object has already been deleted or has not been completely created

javax.xml.ws.soap.SOAPFaultException: The object has already been deleted or has not been completely created
        at com.sun.xml.internal.ws.fault.SOAP11Fault.getProtocolException(SOAP11Fault.java:178)
        at com.sun.xml.internal.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:119)
        at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:108)
        at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:78)
        at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(SEIStub.java:110)
        at com.sun.proxy.$Proxy389.retrieveProperties(Unknown Source)
        at com.cloud.hypervisor.vmware.util.VmwareClient.retrieveMoRefProperties(VmwareClient.java:308)
        at com.cloud.hypervisor.vmware.util.VmwareClient.getDynamicProperty(VmwareClient.java:263)
        at com.cloud.hypervisor.vmware.mo.HostMO.getHostDatastoreSystemMO(HostMO.java:166)
        at com.cloud.hypervisor.vmware.mo.HostMO.findDatastore(HostMO.java:889)
        at com.cloud.hypervisor.vmware.mo.HypervisorHostHelper.findDatastoreWithBackwardsCompatibility(HypervisorHostHelper.java:131)
        at com.cloud.storage.resource.VmwareStorageProcessor.deleteVolume(VmwareStorageProcessor.java:1481)
        at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:120)
        at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:54)
        at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:598)
        at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:215)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:50)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:47)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:701)
{noformat}

+Root Cause Analysis+
The thread local context which is used to handle connections to a vCenter is being re-used intermittently. And when it is re-used, if the existing context corresponds to the other vCenter then the operation being executed will fail.
The re-use is happening because after the PingTask completion we are not clearing the ThreadLocal context.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)