You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@jclouds.apache.org by "Yaron Rosenbaum (JIRA)" <ji...@apache.org> on 2014/11/20 08:14:33 UTC
[jira] [Commented] (JCLOUDS-767) Failed spot request but still
created, node is null - unable to cancel
[ https://issues.apache.org/jira/browse/JCLOUDS-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14219079#comment-14219079 ]
Yaron Rosenbaum commented on JCLOUDS-767:
-----------------------------------------
Hi
After further research with Amazon, it looks like jClouds is doing some sort of POLLING on spot requests, to check if they have been fulfilled and/or the instance has been created. It looks like, this polling happens too frequently for Amazon, and they perceive this as some sort of attack, and therefore send the error I have reported.
I would like to suggest:
1) Increasing the polling interval,
2) Sending a new polling request only when the previous one has returned (or a long timeout has ben breached)
3) Possibly allowing this to be a configurable parameter
At the moment, the result of this behaviour is that no more than one concurrent spot request may be handled by jClouds.
Note, that I am requesting a number of spots on different threads, and not issuing one request for multiple spots.
Here's what Amazon support say:
"This error comes from querying bid status (e.g., "ec2-describe-spot-instance-requests") too frequently. In this case, you are calling this API before a previous call returned (perhaps calling from multiple clients in a tight loop). While that's supported by many AWS APIs, we've had problems with brown-outs because of it and need to protect the service. We believe that most cases where a customer is throttled will be avoided by simply waiting for a response before calling again (sleeping for 1 second between response received and next request would avoid throttling for most Spot customers). The underlying bid data changes slowly, so there little to be gained by calling more than twice a minute in any case.
"
> Failed spot request but still created, node is null - unable to cancel
> ----------------------------------------------------------------------
>
> Key: JCLOUDS-767
> URL: https://issues.apache.org/jira/browse/JCLOUDS-767
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-compute
> Affects Versions: 1.8.1
> Environment: AWS, Spot instances, CoreOS
> Reporter: Yaron Rosenbaum
> Priority: Critical
>
> My program creates spot instances (see 'addNode' below).
> Recently, in some cases I've been getting "RequestResourceCountExceeded" error (see below).
> The problem is that JClouds throws an exception, but the spot request IS created and executed by AWS. The node is never properly configured. Thus, I have no way to cancel the spot request (there's no node ID).
> Correct behaviour should be: Failure should leave no active spot requests.
> At the very least, there should be a way to cancel the spot request if it fails.
> Also, it does seem like it shouldn't have failed - since the spot request is successful...
> public ComputeMetadata addNode(Template template) throws RunNodesException {
> ComputeMetadata result = Iterables.getOnlyElement(getComputeService().createNodesInGroup(getGroupName(), 1, template));
> LOG.info("Node \"{}\" created", result.getName());
> if (isLoadBalancerLogicEnabled()) {
> LOG.info("Adding node \"{}\" to load balancer", result.getName());
> addNodeToLoadBalancer(result);
> }
> 10:05:43.966 [main] INFO io.multicloud.controller.AWSProvider - Successfully created compute context
> 10:06:02.795 [main] DEBUG i.m.controller.CloudProviderImpl - Current nodes:
> 10:06:02.795 [main] INFO io.multicloud.controller.AWSProvider - quorum is missing 3 nodes
> 10:06:03.077 [main] INFO i.m.controller.CloudProviderImpl - Adding 3 nodes
> 10:06:04.113 [pool-3-thread-1] WARN jclouds.compute - << not api visible instances([{region=us-east-1, name=sir-02gebgm2}])
> 10:06:04.342 [pool-3-thread-3] WARN jclouds.compute - << not api visible instances([{region=us-east-1, name=sir-02gez2h5}])
> 10:06:04.460 [pool-3-thread-2] WARN jclouds.compute - << not api visible instances([{region=us-east-1, name=sir-02ghn9t2}])
> 10:06:17.853 [user thread 1] ERROR jclouds.compute - << problem customizing node(us-east-1/sir-02gez2h5):
> org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='c5ffce45-02c3-4443-b4c2-42cfe3e2a7ef', requestToken='null', code='RequestResourceCountExceeded', message='the amount of data you requested and frequency of requests together exceed the levels allowed by the Spot service', context='{Response=, Errors=}'}
> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75) ~[sts-1.8.1.jar:1.8.1]
> at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65) ~[jclouds-core-1.8.1.jar:1.8.1]
> at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128) ~[jclouds-core-1.8.1.jar:1.8.1]
> at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98) ~[jclouds-core-1.8.1.jar:1.8.1]
> at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90) ~[jclouds-core-1.8.1.jar:1.8.1]
> at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73) ~[jclouds-core-1.8.1.jar:1.8.1]
> at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44) ~[jclouds-core-1.8.1.jar:1.8.1]
> at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117) ~[jclouds-core-1.8.1.jar:1.8.1]
> at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87) ~[guava-17.0.jar:na]
> at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source) ~[na:na]
> at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53) ~[aws-ec2-1.8.1.jar:1.8.1]
> at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56) ~[ec2-1.8.1.jar:1.8.1]
> at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58) ~[jclouds-compute-1.8.1.jar:1.8.1]
> at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38) ~[jclouds-compute-1.8.1.jar:1.8.1]
> at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57) ~[jclouds-compute-1.8.1.jar:1.8.1]
> at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40) ~[jclouds-compute-1.8.1.jar:1.8.1]
> at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142) ~[jclouds-compute-1.8.1.jar:1.8.1]
> at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137) ~[jclouds-compute-1.8.1.jar:1.8.1]
> at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111) ~[jclouds-core-1.8.1.jar:1.8.1]
> at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154) ~[jclouds-compute-1.8.1.jar:1.8.1]
> at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118) ~[jclouds-compute-1.8.1.jar:1.8.1]
> at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63) ~[jclouds-compute-1.8.1.jar:1.8.1]
> at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42) ~[jclouds-compute-1.8.1.jar:1.8.1]
> at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117) [jclouds-compute-1.8.1.jar:1.8.1]
> at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49) [jclouds-compute-1.8.1.jar:1.8.1]
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [na:1.7.0_09]
> at java.util.concurrent.FutureTask.run(FutureTask.java:166) [na:1.7.0_09]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_09]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_09]
> at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
> 10:06:17.872 [pool-3-thread-3] ERROR i.m.controller.CloudProviderImpl - Failed to create node:
> org.jclouds.compute.RunNodesException: error running 1 node group(AWS-East-1) location(us-east-1) image(ami-06ff446e) size(m1.medium) options({userMetadata={TYPE=SPOT}, keyPair=Yaron, userDataCksum=c242210b, subnetId=subnet-4f5a433b, spotPrice=0.013, groupIds=[sg-ce6512ab, sg-d9cb77bc]})
> Execution failures:
> 0 error[s]
> Node failures:
> 1) AWSResponseException on node us-east-1/sir-02gez2h5:
> org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='c5ffce45-02c3-4443-b4c2-42cfe3e2a7ef', requestToken='null', code='RequestResourceCountExceeded', message='the amount of data you requested and frequency of requests together exceed the levels allowed by the Spot service', context='{Response=, Errors=}'}
> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
> at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
> at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
> at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
> at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
> at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
> at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
> at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
> at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
> at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
> at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
> at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
> at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
> at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
> at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
> at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
> at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
> at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
> at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 1 error[s]
> at org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:222) ~[jclouds-compute-1.8.1.jar:1.8.1]
> at org.jclouds.ec2.compute.EC2ComputeService.createNodesInGroup(EC2ComputeService.java:148) ~[ec2-1.8.1.jar:1.8.1]
> at io.multicloud.controller.CloudProviderImpl.addNode(CloudProviderImpl.java:65) ~[classes/:na]
> at io.multicloud.controller.CloudProviderImpl$CreateNewNodeTask.run(CloudProviderImpl.java:227) ~[classes/:na]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_09]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_09]
> at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
> 10:07:47.218 [user thread 0] ERROR jclouds.compute - << problem customizing node(us-east-1/sir-02gebgm2):
> org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='82736b80-b101-4111-acd0-69e3ce53d645', requestToken='null', code='RequestResourceCountExceeded', message='the amount of data you requested and frequency of requests together exceed the levels allowed by the Spot service', context='{Response=, Errors=}'}
> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75) ~[sts-1.8.1.jar:1.8.1]
> at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65) ~[jclouds-core-1.8.1.jar:1.8.1]
> at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128) ~[jclouds-core-1.8.1.jar:1.8.1]
> at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98) ~[jclouds-core-1.8.1.jar:1.8.1]
> at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90) ~[jclouds-core-1.8.1.jar:1.8.1]
> at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73) ~[jclouds-core-1.8.1.jar:1.8.1]
> at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44) ~[jclouds-core-1.8.1.jar:1.8.1]
> at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117) ~[jclouds-core-1.8.1.jar:1.8.1]
> at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87) ~[guava-17.0.jar:na]
> at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source) ~[na:na]
> at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53) ~[aws-ec2-1.8.1.jar:1.8.1]
> at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56) ~[ec2-1.8.1.jar:1.8.1]
> at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58) ~[jclouds-compute-1.8.1.jar:1.8.1]
> at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38) ~[jclouds-compute-1.8.1.jar:1.8.1]
> at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57) ~[jclouds-compute-1.8.1.jar:1.8.1]
> at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40) ~[jclouds-compute-1.8.1.jar:1.8.1]
> at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142) ~[jclouds-compute-1.8.1.jar:1.8.1]
> at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137) ~[jclouds-compute-1.8.1.jar:1.8.1]
> at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111) ~[jclouds-core-1.8.1.jar:1.8.1]
> at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154) ~[jclouds-compute-1.8.1.jar:1.8.1]
> at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118) ~[jclouds-compute-1.8.1.jar:1.8.1]
> at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63) ~[jclouds-compute-1.8.1.jar:1.8.1]
> at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42) ~[jclouds-compute-1.8.1.jar:1.8.1]
> at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117) [jclouds-compute-1.8.1.jar:1.8.1]
> at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49) [jclouds-compute-1.8.1.jar:1.8.1]
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [na:1.7.0_09]
> at java.util.concurrent.FutureTask.run(FutureTask.java:166) [na:1.7.0_09]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_09]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_09]
> at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
> 10:07:47.220 [pool-3-thread-1] ERROR i.m.controller.CloudProviderImpl - Failed to create node:
> org.jclouds.compute.RunNodesException: error running 1 node group(AWS-East-1) location(us-east-1) image(ami-06ff446e) size(m1.medium) options({userMetadata={TYPE=SPOT}, keyPair=Yaron, userDataCksum=c242210b, subnetId=subnet-0afda622, spotPrice=0.013, groupIds=[sg-ce6512ab, sg-d9cb77bc]})
> Execution failures:
> 0 error[s]
> Node failures:
> 1) AWSResponseException on node us-east-1/sir-02gebgm2:
> org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='82736b80-b101-4111-acd0-69e3ce53d645', requestToken='null', code='RequestResourceCountExceeded', message='the amount of data you requested and frequency of requests together exceed the levels allowed by the Spot service', context='{Response=, Errors=}'}
> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
> at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
> at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
> at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
> at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
> at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
> at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
> at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
> at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
> at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
> at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
> at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
> at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
> at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
> at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
> at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
> at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
> at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
> at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 1 error[s]
> at org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:222) ~[jclouds-compute-1.8.1.jar:1.8.1]
> at org.jclouds.ec2.compute.EC2ComputeService.createNodesInGroup(EC2ComputeService.java:148) ~[ec2-1.8.1.jar:1.8.1]
> at io.multicloud.controller.CloudProviderImpl.addNode(CloudProviderImpl.java:65) ~[classes/:na]
> at io.multicloud.controller.CloudProviderImpl$CreateNewNodeTask.run(CloudProviderImpl.java:227) ~[classes/:na]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_09]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_09]
> at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
> 10:09:24.051 [pool-3-thread-2] INFO i.m.controller.CloudProviderImpl - Node "AWS-East-1-8b5f5f66" created
> 10:09:24.051 [pool-3-thread-2] INFO i.m.controller.CloudProviderImpl - Adding node "AWS-East-1-8b5f5f66" to load balancer
> 10:09:25.110 [pool-3-thread-2] INFO i.m.controller.CloudProviderImpl - Creating node using template: {image={id=us-east-1/ami-06ff446e, providerId=ami-06ff446e, name=CoreOS-beta-444.5.0, location={scope=REGION, id=us-east-1, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA]}, os={family=unrecognized, arch=paravirtual, version=, description=595879546273/CoreOS-beta-444.5.0, is64Bit=true}, description=CoreOS beta 444.5.0 (PV), status=AVAILABLE[available], loginUser=root, userMetadata={owner=595879546273, rootDeviceType=ebs, virtualizationType=paravirtual, hypervisor=xen}}, hardware={id=m1.medium, providerId=m1.medium, processors=[{cores=1.0, speed=2.0}], ram=3750, volumes=[{type=LOCAL, size=10.0, device=/dev/sda1, bootDevice=true, durable=false}, {type=LOCAL, size=420.0, device=/dev/sdb, bootDevice=false, durable=false}, {type=LOCAL, size=420.0, device=/dev/sdc, bootDevice=false, durable=false}], supportsImage=Predicates.and(Predicates.alwaysTrue(),Predicates.or(isWindows(),requiresVirtualizationType(paravirtual)),Predicates.alwaysTrue(),Predicates.alwaysTrue())}, location={scope=REGION, id=us-east-1, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA]}, options={userMetadata={TYPE=SPOT}, keyPair=Yaron, userDataCksum=c242210b, subnetId=subnet-50cdf616, spotPrice=0.013, groupIds=[sg-ce6512ab, sg-d9cb77bc]}}
> 10:09:39.105 [main] DEBUG i.m.controller.CloudProviderImpl - Current nodes:
> 10:09:39.105 [main] DEBUG i.m.controller.CloudProviderImpl - Name: AWS-East-1-8b5f5f66, User metadata: {TYPE=SPOT, Name=AWS-East-1-8b5f5f66}
> 10:09:39.106 [main] INFO io.multicloud.controller.AWSProvider - Optimizing availability: Adding 1 nodes
> 10:09:39.106 [main] INFO i.m.controller.CloudProviderImpl - Adding 1 nodes
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)