You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by "Aled Sage (JIRA)" <ji...@apache.org> on 2017/09/18 08:40:00 UTC
[jira] [Created] (BROOKLYN-534) ServiceRestarter policy provisioned
new VM while entity was stopping
Aled Sage created BROOKLYN-534:
----------------------------------
Summary: ServiceRestarter policy provisioned new VM while entity was stopping
Key: BROOKLYN-534
URL: https://issues.apache.org/jira/browse/BROOKLYN-534
Project: Brooklyn
Issue Type: Bug
Affects Versions: 0.11.0
Reporter: Aled Sage
I had an app consisting of approx 100 VMs. I called stop, to terminate all the VMs.
However, for one of the entities (nginx) it failed to stop the VM (see https://issues.apache.org/jira/browse/BROOKLYN-533). Unfortunately the {{ServiceFailureDetector}} was still active, which caused the {{ServiceRestarter}} to try to restart it! Because we'd partially shutdown nginx, it decided that a new machine needed to be provisioned!
I presume that a fix would be to change {{ServiceRestarter}} so that it doesn't try to restart it if the expected state is 'stopping' or 'stopped'. We should also check other policies (e.g. {{ServiceReplacer}} to ensure they don't kick in if their associated entity is being stopped.
Snippets from the log are shown below:
{noformat}
2017-09-15T17:32:52,268 INFO 131 o.a.b.e.s.b.l.MachineLifecycleEffectorTasks [r-VlI23lev-80356] Stopping NginxControllerImpl{id=t2ta5sv8ti} in [SshMachineLocation[34.252.178.167:aled@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22(id=taep3uro9m)]]
2017-09-15T17:32:52,599 DEBUG 122 b.SSH [r-VlI23lev-80528] stopping NginxControllerImpl{id=t2ta5sv8ti}, initiating ssh on machine SshMachineLocation[34.252.178.167:aled@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22(id=taep3uro9m)]: #!/bin/bash -e
2017-09-15T17:34:09,965 ERROR 127 o.a.b.l.j.JcloudsLocation [r-VlI23lev-80548] Problem releasing machine SshMachineLocation[34.252.178.167:aled@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22(id=taep3uro9m)] in JcloudsLocation[AWS Dublin:xxxxxxxx@xxxxxxxx], instance id eu-west-1/i-0663997ccc85af459; ignoring and continuing, will throw subsequently: org.jclouds.aws.AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error: AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2', requestToken='null', code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=, Errors=}'}
org.jclouds.aws.AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error: AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2', requestToken='null', code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=, Errors=}'}
at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75) [259:sts:2.0.2]
at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:67) [101:jclouds-core:2.0.2.2-20170712_1657]
at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:140) [101:jclouds-core:2.0.2.2-20170712_1657]
at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:109) [101:jclouds-core:2.0.2.2-20170712_1657]
at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90) [101:jclouds-core:2.0.2.2-20170712_1657]
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73) [101:jclouds-core:2.0.2.2-20170712_1657]
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44) [101:jclouds-core:2.0.2.2-20170712_1657]
at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117) [101:jclouds-core:2.0.2.2-20170712_1657]
at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87) [66:com.google.guava:18.0.0]
at com.sun.proxy.$Proxy179.describeInstancesInRegion(Unknown Source) [47:aws-ec2:2.0.2]
at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getRunningInstanceInRegion(EC2GetNodeMetadataStrategy.java:64) [77:ec2:2.0.2]
at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:52) [47:aws-ec2:2.0.2]
at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56) [77:ec2:2.0.2]
at org.jclouds.compute.predicates.AtomicNodeTerminated.refreshOrNull(AtomicNodeTerminated.java:42) [100:jclouds-compute:2.0.2]
at org.jclouds.compute.predicates.AtomicNodeTerminated.refreshOrNull(AtomicNodeTerminated.java:28) [100:jclouds-compute:2.0.2]
at org.jclouds.compute.predicates.internal.TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.apply(TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.java:46) [100:jclouds-compute:2.0.2]
at org.jclouds.compute.predicates.internal.TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.apply(TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.java:31) [100:jclouds-compute:2.0.2]
at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:117) [101:jclouds-core:2.0.2.2-20170712_1657]
at org.jclouds.compute.internal.BaseComputeService.doDestroyNode(BaseComputeService.java:309) [100:jclouds-compute:2.0.2]
at org.jclouds.compute.internal.BaseComputeService.destroyNode(BaseComputeService.java:250) [100:jclouds-compute:2.0.2]
at org.apache.brooklyn.location.jclouds.JcloudsLocation.releaseNode(JcloudsLocation.java:2189) [127:org.apache.brooklyn.locations-jclouds:0.12.0.SNAPSHOT]
at org.apache.brooklyn.location.jclouds.JcloudsLocation.release(JcloudsLocation.java:2141) [127:org.apache.brooklyn.locations-jclouds:0.12.0.SNAPSHOT]
at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.stopAnyProvisionedMachines(MachineLifecycleEffectorTasks.java:1033) [131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:883) [131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:880) [131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
at org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:363) [122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:529) [122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:?]
at java.lang.Thread.run(Thread.java:748) [?:?]
2017-09-15T17:34:09,966 DEBUG 122 o.a.b.l.s.SshMachineLocation [r-VlI23lev-80548] SshMachineLocation[34.252.178.167:aled@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22(id=taep3uro9m)] invalidating all entries in ssh pool cache. Final stats: CacheStats{hitCount=803, missCount=6, loadSuccessCount=6, loadExceptionCount=0, totalLoadTime=1248584, evictionCount=4}
2017-09-15T17:34:09,966 DEBUG 122 o.a.b.l.s.SshMachineLocation [r-VlI23lev-80548] BasicPool{name=Pool(34.252.178.167@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22:hash1856793534)} evicted from SshTool cache. Eviction cause: EXPLICIT
2017-09-15T17:34:09,966 DEBUG 143 o.a.b.u.p.BasicPool [r-VlI23lev-80548] BasicPool{name=Pool(34.252.178.167@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22:hash1856793534)} closing, with 1 resources (currentLeased=0; totalLeased=789; totalCreated=1; totalClosed=0)
2017-09-15T17:34:09,966 DEBUG 122 o.a.b.l.s.SshMachineLocation [r-VlI23lev-80548] org.apache.brooklyn.location.ssh.SshMachineLocation$8@710df8f2 closing pool for aled@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com:22
2017-09-15T17:34:09,966 DEBUG 122 o.a.b.l.s.SshMachineLocation [r-VlI23lev-80548] BasicPool{name=Pool(34.252.178.167@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22:hash1856793534)} evicted from SshTool cache. Eviction cause: EXPLICIT
2017-09-15T17:34:09,966 DEBUG 143 o.a.b.u.p.BasicPool [r-VlI23lev-80548] BasicPool{name=Pool(34.252.178.167@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22:hash1856793534)} closing, with 1 resources (currentLeased=0; totalLeased=1; totalCreated=1; totalClosed=0)
2017-09-15T17:34:09,966 DEBUG 122 o.a.b.l.s.SshMachineLocation [r-VlI23lev-80548] org.apache.brooklyn.location.ssh.SshMachineLocation$8@6a0ceb61 closing pool for aled@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com:22
2017-09-15T17:34:09,967 DEBUG 122 o.a.b.c.m.i.LocalLocationManager [r-VlI23lev-80548] org.apache.brooklyn.core.mgmt.internal.LocalLocationManager@68f8b22f stopped management of location SshMachineLocation[34.252.178.167:aled@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22(id=taep3uro9m)]
2017-09-15T17:34:09,967 DEBUG 122 o.a.b.c.m.i.LocalLocationManager [r-VlI23lev-80548] Missing transition mode for SshMachineLocation[34.252.178.167:aled@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22(id=taep3uro9m)] when unmanaging; assuming primary/destroying
2017-09-15T17:34:09,967 DEBUG 122 o.a.b.c.m.i.LocalUsageManager [r-VlI23lev-80548] Storing location lifecycle usage event: location SshMachineLocation[34.252.178.167:aled@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22(id=taep3uro9m)] in state destroyed; caller context NginxControllerImpl{id=t2ta5sv8ti}
2017-09-15T17:34:09,967 INFO 127 o.a.b.l.j.JcloudsLocation [r-VlI23lev-80548] Attempted release of unknown machine SshMachineLocation[34.252.178.167:aled@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22(id=taep3uro9m)] in JcloudsLocation[AWS Dublin:xxxxxxxx@xxxxxxxx]
2017-09-15T17:34:09,967 DEBUG 122 o.a.b.c.m.i.LocalLocationManager [r-VlI23lev-80548] Error releasing SshMachineLocation[34.252.178.167:aled@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22(id=taep3uro9m)] in its parent JcloudsLocation[AWS Dublin:xxxxxxxx@xxxxxxxx]: java.lang.IllegalArgumentException: Unknown machine SshMachineLocation[34.252.178.167:aled@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22(id=taep3uro9m)]
2017-09-15T17:34:09,967 DEBUG 122 o.a.b.c.s.AttributeMap [r-VlI23lev-80548] removing attribute internal.termination.task.state on NginxControllerImpl{id=t2ta5sv8ti}
2017-09-15T17:34:09,968 DEBUG 122 o.a.b.c.e.AbstractEntity [r-VlI23lev-80548] Emitting sensor notification entity.sensor.removed value Sensor: internal.termination.task.state (org.apache.brooklyn.core.entity.internal.AttributesInternal$ProvisioningTaskState) on NginxControllerImpl{id=t2ta5sv8ti}
...
2017-09-15T17:34:09,972 DEBUG 129 o.a.b.p.h.ServiceFailureDetector [r-VlI23lev-80386] ServiceFailureDetector{uniqueTag=service.state.actual, running=true, entity=NginxControllerImpl{id=t2ta5sv8ti}, id=eox0ce3omo} health-check for NginxControllerImpl{id=t2ta5sv8ti}, component now failing: location=[]; status=Present[value=on-fire]; lastPublished=NONE; timeNow=2017-09-15 17:34:09.972; currentFailurePeriod=<none> (stabilization 0ms); currentRecoveryPeriod=<none> (stabilization 0ms)
2017-09-15T17:34:09,973 DEBUG 129 o.a.b.p.h.ServiceFailureDetector [r-VlI23lev-80386] ServiceFailureDetector{uniqueTag=service.state.actual, running=true, entity=NginxControllerImpl{id=t2ta5sv8ti}, id=eox0ce3omo} publishing failed (state=Present[value=on-fire]; currentFailureStartTime=2017-09-15 17:34:09.972; now=2017-09-15 17:34:09.972
2017-09-15T17:34:09,973 DEBUG 122 o.a.b.c.e.AbstractEntity [r-VlI23lev-80386] Emitting sensor notification ha.entityFailed value FailureDescriptor{component=NginxControllerImpl{id=t2ta5sv8ti}, description=service not up} on NginxControllerImpl{id=t2ta5sv8ti}
2017-09-15T17:34:09,973 DEBUG 129 o.a.b.p.h.ServiceFailureDetector [r-VlI23lev-80386] ServiceFailureDetector{uniqueTag=service.state.actual, running=true, entity=NginxControllerImpl{id=t2ta5sv8ti}, id=eox0ce3omo} setting on-fire, now that deferred period has passed (state=Present[value=on-fire])
2017-09-15T17:34:09,973 INFO 129 o.a.b.p.h.ServiceRestarter [r-VlI23lev-80803] ServiceRestarter notified; dispatching job for NginxControllerImpl{id=t2ta5sv8ti} (FailureDescriptor{component=NginxControllerImpl{id=t2ta5sv8ti}, description=service not up})
2017-09-15T17:34:09,974 WARN 129 o.a.b.p.h.ServiceRestarter [r-VlI23lev-80548] ServiceRestarter acting on failure detected at NginxControllerImpl{id=t2ta5sv8ti} (FailureDescriptor{component=NginxControllerImpl{id=t2ta5sv8ti}, description=service not up})
2017-09-15T17:34:09,974 DEBUG 129 o.a.b.p.h.ServiceFailureDetector [r-VlI23lev-80356] ServiceFailureDetector{uniqueTag=service.state.actual, running=true, entity=NginxControllerImpl{id=t2ta5sv8ti}, id=eox0ce3omo} setting on-fire, now that deferred period has passed (state=Present[value=on-fire])
2017-09-15T17:34:09,974 DEBUG 122 o.a.b.u.c.t.CompoundTask [r-VlI23lev-80356] Parent task DstJob:pXCtsb5q ignoring child error (org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error: AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2', requestToken='null', code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=, Errors=}'}) in presence of our own error (org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error: AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2', requestToken='null', code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=, Errors=}'})
2017-09-15T17:34:09,974 DEBUG 122 o.a.b.c.e.Effectors [r-VlI23lev-80548] Replacing invocation of restart[] on NginxControllerImpl{id=t2ta5sv8ti} with restart[restartChildren,restartMachine] which is the impl defined at that entity
2017-09-15T17:34:09,974 WARN 122 o.a.b.c.m.i.EffectorUtils [r-VlI23lev-80356] Error invoking stop at NginxControllerImpl{id=t2ta5sv8ti}: AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error: AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2', requestToken='null', code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=, Errors=}'}
2017-09-15T17:34:09,975 DEBUG 131 o.a.b.e.s.b.SoftwareProcessDriverLifecycleEffectorTasks [r-VlI23lev-80386] restart of NginxControllerImpl{id=t2ta5sv8ti} has no hostname - doing machine-level restart
2017-09-15T17:34:09,975 DEBUG 131 o.a.b.e.s.b.SoftwareProcessDriverLifecycleEffectorTasks [r-VlI23lev-80386] restart of NginxControllerImpl{id=t2ta5sv8ti} requested be applied at machine level
2017-09-15T17:34:09,975 DEBUG 131 o.a.b.e.s.b.SoftwareProcessDriverLifecycleEffectorTasks [r-VlI23lev-80386] restart of NginxControllerImpl{id=t2ta5sv8ti} has no hostname - doing machine-level restart
...
2017-09-15T17:34:09,975 DEBUG 122 o.a.b.c.m.i.EffectorUtils [r-VlI23lev-80356] Error invoking stop at NginxControllerImpl{id=t2ta5sv8ti}
org.apache.brooklyn.util.exceptions.PropagatedRuntimeException:
at org.apache.brooklyn.util.exceptions.Exceptions.propagate(Exceptions.java:129) [143:org.apache.brooklyn.utils-common:0.12.0.SNAPSHOT]
at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.doStop(MachineLifecycleEffectorTasks.java:870) [131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.doStopLatching(MachineLifecycleEffectorTasks.java:757) [131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.stop(MachineLifecycleEffectorTasks.java:744) [131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopEffectorBody.call(MachineLifecycleEffectorTasks.java:293) [131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopEffectorBody.call(MachineLifecycleEffectorTasks.java:290) [131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
at org.apache.brooklyn.core.effector.EffectorTasks$EffectorBodyTaskFactory$1.call(EffectorTasks.java:82) [122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
at org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:363) [122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:529) [122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:?]
at java.lang.Thread.run(Thread.java:748) [?:?]
Caused by: java.util.concurrent.ExecutionException: org.jclouds.aws.AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error: AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2', requestToken='null', code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=, Errors=}'}
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]
at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:?]
at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:63) ~[?:?]
at org.apache.brooklyn.util.core.task.BasicTask.get(BasicTask.java:382) ~[?:?]
at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.doStop(MachineLifecycleEffectorTasks.java:858) ~[?:?]
... 11 more
Caused by: org.jclouds.aws.AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error: AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2', requestToken='null', code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=, Errors=}'}
at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75) ~[?:?]
at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:67) ~[?:?]
at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:140) ~[?:?]
at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:109) ~[?:?]
at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90) ~[?:?]
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73) ~[?:?]
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44) ~[?:?]
at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117) ~[?:?]
at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87) ~[?:?]
at com.sun.proxy.$Proxy179.describeInstancesInRegion(Unknown Source) ~[?:?]
at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getRunningInstanceInRegion(EC2GetNodeMetadataStrategy.java:64) ~[?:?]
at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:52) ~[?:?]
at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56) ~[?:?]
at org.jclouds.compute.predicates.AtomicNodeTerminated.refreshOrNull(AtomicNodeTerminated.java:42) ~[?:?]
at org.jclouds.compute.predicates.AtomicNodeTerminated.refreshOrNull(AtomicNodeTerminated.java:28) ~[?:?]
at org.jclouds.compute.predicates.internal.TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.apply(TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.java:46) ~[?:?]
at org.jclouds.compute.predicates.internal.TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.apply(TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.java:31) ~[?:?]
at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:117) ~[?:?]
at org.jclouds.compute.internal.BaseComputeService.doDestroyNode(BaseComputeService.java:309) ~[?:?]
at org.jclouds.compute.internal.BaseComputeService.destroyNode(BaseComputeService.java:250) ~[?:?]
at org.apache.brooklyn.location.jclouds.JcloudsLocation.releaseNode(JcloudsLocation.java:2189) ~[?:?]
at org.apache.brooklyn.location.jclouds.JcloudsLocation.release(JcloudsLocation.java:2141) ~[?:?]
at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.stopAnyProvisionedMachines(MachineLifecycleEffectorTasks.java:1033) ~[?:?]
at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:883) ~[?:?]
at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:880) ~[?:?]
... 6 more
2017-09-15T17:34:09,978 DEBUG 122 o.a.b.u.c.t.BasicExecutionManager [r-VlI23lev-80356] Exception running task Task[stop]@pXCtsb5q (rethrowing): org.apache.brooklyn.core.mgmt.internal.EffectorUtils$EffectorCallPropagatedRuntimeException: Error invoking stop at NginxControllerImpl{id=t2ta5sv8ti}: AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error: AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2', requestToken='null', code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=, Errors=}'}
2017-09-15T17:34:09,979 INFO 131 o.a.b.e.s.b.l.MachineLifecycleEffectorTasks [r-VlI23lev-80833] Stopping NginxControllerImpl{id=t2ta5sv8ti} in []
...
2017-09-15T17:34:10,262 ERROR 107 o.j.a.h.AWSServerErrorRetryHandler [r-VlI23lev-81233] Cannot retry after server error, command has exceeded retry limit 6: [method=org.jclouds.aws.ec2.features.AWSInstanceApi.public abstract java.util.Set org.jclouds.aws.ec2.features.AWSInstanceApi.describeInstancesInRegion(java.lang.String,java.lang.String[])[eu-west-1, [Ljava.lang.String;@3c00d5d0], request=POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1]
2017-09-15T17:34:10,263 ERROR 127 o.a.b.l.j.JcloudsLocation [r-VlI23lev-81233] Problem releasing machine SshMachineLocation[34.240.150.69:aled@ec2-34-240-150-69.eu-west-1.compute.amazonaws.com/34.240.150.69:22(id=dpxjea986f)] in JcloudsLocation[AWS Dublin:xxxxxxxx@xxxxxxxx], instance id eu-west-1/i-02e2311ab1f16a0c8; ignoring and continuing, will throw subsequently: org.jclouds.aws.AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error: AWSError{requestId='e0e6ebbf-9a19-43ab-9219-25907eb8aa71', requestToken='null', code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=, Errors=}'}
org.jclouds.aws.AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error: AWSError{requestId='e0e6ebbf-9a19-43ab-9219-25907eb8aa71', requestToken='null', code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=, Errors=}'}
at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75) [259:sts:2.0.2]
at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:67) [101:jclouds-core:2.0.2.2-20170712_1657]
at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:140) [101:jclouds-core:2.0.2.2-20170712_1657]
at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:109) [101:jclouds-core:2.0.2.2-20170712_1657]
at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90) [101:jclouds-core:2.0.2.2-20170712_1657]
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73) [101:jclouds-core:2.0.2.2-20170712_1657]
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44) [101:jclouds-core:2.0.2.2-20170712_1657]
at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117) [101:jclouds-core:2.0.2.2-20170712_1657]
at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87) [66:com.google.guava:18.0.0]
at com.sun.proxy.$Proxy179.describeInstancesInRegion(Unknown Source) [47:aws-ec2:2.0.2]
at org.jclouds.aws.ec2.compute.strategy.AWSEC2DestroyNodeStrategy.destroyInstanceInRegion(AWSEC2DestroyNodeStrategy.java:59) [47:aws-ec2:2.0.2]
at org.jclouds.ec2.compute.strategy.EC2DestroyNodeStrategy.destroyNode(EC2DestroyNodeStrategy.java:72) [77:ec2:2.0.2]
at org.jclouds.compute.internal.BaseComputeService$2.apply(BaseComputeService.java:298) [100:jclouds-compute:2.0.2]
at org.jclouds.compute.internal.BaseComputeService$2.apply(BaseComputeService.java:295) [100:jclouds-compute:2.0.2]
at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:117) [101:jclouds-core:2.0.2.2-20170712_1657]
at org.jclouds.compute.internal.BaseComputeService.doDestroyNode(BaseComputeService.java:309) [100:jclouds-compute:2.0.2]
at org.jclouds.compute.internal.BaseComputeService.destroyNode(BaseComputeService.java:250) [100:jclouds-compute:2.0.2]
at org.apache.brooklyn.location.jclouds.JcloudsLocation.releaseNode(JcloudsLocation.java:2189) [127:org.apache.brooklyn.locations-jclouds:0.12.0.SNAPSHOT]
at org.apache.brooklyn.location.jclouds.JcloudsLocation.release(JcloudsLocation.java:2141) [127:org.apache.brooklyn.locations-jclouds:0.12.0.SNAPSHOT]
at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.stopAnyProvisionedMachines(MachineLifecycleEffectorTasks.java:1033) [131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:883) [131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:880) [131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
at org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:363) [122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:529) [122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:?]
at java.lang.Thread.run(Thread.java:748) [?:?]
...
2017-09-15T17:36:28,127 INFO 127 o.a.b.l.j.JcloudsLocation [r-VlI23lev-81071] Creating VM aws-ec2:eu-west-1@NginxControllerImpl{id=h0xas9pjk5} in JcloudsLocation[AWS Dublin:xxxxxxxx@xxxxxxxx]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)