You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@deltacloud.apache.org by "wes hayutin (JIRA)" <ji...@apache.org> on 2011/05/31 21:01:48 UTC
[jira] [Created] (DTACLOUD-47) Request limit exceeded." response
from ec2, get tracebacks in deltacloud log
Request limit exceeded." response from ec2, get tracebacks in deltacloud log
----------------------------------------------------------------------------
Key: DTACLOUD-47
URL: https://issues.apache.org/jira/browse/DTACLOUD-47
Project: DeltaCloud
Issue Type: Bug
Components: Client (Ruby)
Environment: ec2 + aeolus
Reporter: wes hayutin
Assignee: David Lutterkort
Under heavy use, starting 15-20 instances serially its possible to get a response back from ec2 that the "request limit has been reached". Once a load is created, and ec2 can not handle more requests I see a lot of tracebacks in the ec2 log.
127.0.0.1 - - [31/May/2011 20:54:32] "GET /api/instances HTTP/1.1" 500 5628 9.4191
127.0.0.1 - - [31/May/2011 20:54:32] "GET /api/instances HTTP/1.1" 200 52194 9.8775
Deltacloud::BackendError - RequestLimitExceeded: Request limit exceeded.
REQUEST=ec2.us-east-1.amazonaws.com:443/?AWSAccessKeyId=AKIAJ557V2EQ&Action=DeleteTags&ResourceId.1=i-fba01795&SignatureMethod=HmacSHA256&SignatureVersion=2&Tag.1.Key=name&Timestamp=2011-06-01T00%3A54%3A32.000Z&Version=2010-08-31&Signature=ZLFgXz2V9aIIyuV9FTHApP4vNnfo8NxFaf6OEhSCJss%3D
REQUEST ID=55502bf8-4814-41a6-933b-fb02b6a21e20 :
././lib/deltacloud/base_driver/base_driver.rb:247:in `safely'
././lib/deltacloud/base_driver/base_driver.rb:245:in `each'
././lib/deltacloud/base_driver/base_driver.rb:245:in `safely'
././lib/deltacloud/drivers/ec2/ec2_driver.rb:174:in `instances'
././lib/deltacloud/helpers/application_helper.rb:80:in `send'
././lib/deltacloud/helpers/application_helper.rb:80:in `filter_all'
/usr/lib/ruby/1.8/benchmark.rb:293:in `measure'
././lib/deltacloud/helpers/application_helper.rb:79:in `filter_all'
././server.rb:383
././lib/sinatra/rabbit.rb:104:in `instance_eval'
././lib/sinatra/rabbit.rb:104:in `GET /api/instances'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:863:in `call'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:863:in `route'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:521:in `instance_eval'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:521:in `route_eval'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:500:in `route!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:497:in `catch'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:497:in `route!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:476:in `each'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:476:in `route!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:601:in `dispatch!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:411:in `call!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `instance_eval'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `invoke'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `catch'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `invoke'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:411:in `call!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:399:in `call'
././lib/sinatra/rack_driver_select.rb:45:in `call'
././lib/sinatra/rack_matrix_params.rb:80:in `call'
././lib/sinatra/rack_runtime.rb:36:in `call'
././lib/sinatra/rack_etag.rb:42:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-accept-0.4.3/lib/rack/accept/context.rb:22:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/methodoverride.rb:24:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/commonlogger.rb:18:in `call'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:979:in `call'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:1005:in `synchronize'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:979:in `call'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/stats.rb:24:in `call'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/stats.rb:33:in `log'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/stats.rb:24:in `call'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/connection.rb:76:in `pre_process'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/connection.rb:74:in `catch'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/connection.rb:74:in `pre_process'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1060:in `call'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1060:in `spawn_threadpool'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1057:in `initialize'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1057:in `new'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1057:in `spawn_threadpool'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1049:in `defer'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/connection.rb:54:in `process'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/connection.rb:42:in `receive_data'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run_machine'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/backends/base.rb:57:in `start'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/server.rb:156:in `start'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/controllers/controller.rb:80:in `start'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/runner.rb:177:in `send'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/runner.rb:177:in `run_command'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/runner.rb:143:in `run!'
/usr/share/deltacloud-core/bin/deltacloudd:124
/usr/bin/deltacloudd:5:in `load'
/usr/bin/deltacloudd:5
127.0.0.1 - - [31/May/2011 20:54:32] "GET /api/instances HTTP/1.1" 500 5628 9.4123
127.0.0.1 - - [31/May/2011 20:54:33] "GET /api/instances HTTP/1.1" 200 52194 9.9548
Deltacloud::BackendError - RequestLimitExceeded: Request limit exceeded.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Created] (DTACLOUD-47) Request limit exceeded." response from ec2, get tracebacks in deltacloud log
Posted by Michal Fojtik <mf...@redhat.com>.
On May 31, 2011, at 9:01 PM, wes hayutin (JIRA) wrote:
Hi,
According to my discussion with guy, who have contacted Amazon about this
issue I know that there are no limits, but after issuing 'too-many' (tm)
requests, Amazon will start throttle using code 500 and you need to repeat
the request again after some unspecified amount of time.
From my point of view there is no problem with DC API, which just forwards
error message from Ec2 to client. I think error message:
Deltacloud::BackendError - RequestLimitExceeded: Request limit exceeded.
is perfectly appropriate to this and I think you will need to count with this
situation and do retry after some sleep:
begin
client.instances()
rescue
sleep(3)
retry
end
-- Michal
> Request limit exceeded." response from ec2, get tracebacks in deltacloud log
> ----------------------------------------------------------------------------
>
> Key: DTACLOUD-47
> URL: https://issues.apache.org/jira/browse/DTACLOUD-47
> Project: DeltaCloud
> Issue Type: Bug
> Components: Client (Ruby)
> Environment: ec2 + aeolus
> Reporter: wes hayutin
> Assignee: David Lutterkort
>
>
> Under heavy use, starting 15-20 instances serially its possible to get a response back from ec2 that the "request limit has been reached". Once a load is created, and ec2 can not handle more requests I see a lot of tracebacks in the ec2 log.
>
> 127.0.0.1 - - [31/May/2011 20:54:32] "GET /api/instances HTTP/1.1" 500 5628 9.4191
> 127.0.0.1 - - [31/May/2011 20:54:32] "GET /api/instances HTTP/1.1" 200 52194 9.8775
> Deltacloud::BackendError - RequestLimitExceeded: Request limit exceeded.
> REQUEST=ec2.us-east-1.amazonaws.com:443/?AWSAccessKeyId=AKIAJ557V2EQ&Action=DeleteTags&ResourceId.1=i-fba01795&SignatureMethod=HmacSHA256&SignatureVersion=2&Tag.1.Key=name&Timestamp=2011-06-01T00%3A54%3A32.000Z&Version=2010-08-31&Signature=ZLFgXz2V9aIIyuV9FTHApP4vNnfo8NxFaf6OEhSCJss%3D
> REQUEST ID=55502bf8-4814-41a6-933b-fb02b6a21e20 :
> ././lib/deltacloud/base_driver/base_driver.rb:247:in `safely'
> ././lib/deltacloud/base_driver/base_driver.rb:245:in `each'
> ././lib/deltacloud/base_driver/base_driver.rb:245:in `safely'
> ././lib/deltacloud/drivers/ec2/ec2_driver.rb:174:in `instances'
> ././lib/deltacloud/helpers/application_helper.rb:80:in `send'
> ././lib/deltacloud/helpers/application_helper.rb:80:in `filter_all'
> /usr/lib/ruby/1.8/benchmark.rb:293:in `measure'
> ././lib/deltacloud/helpers/application_helper.rb:79:in `filter_all'
> ././server.rb:383
> ././lib/sinatra/rabbit.rb:104:in `instance_eval'
> ././lib/sinatra/rabbit.rb:104:in `GET /api/instances'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:863:in `call'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:863:in `route'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:521:in `instance_eval'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:521:in `route_eval'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:500:in `route!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:497:in `catch'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:497:in `route!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:476:in `each'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:476:in `route!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:601:in `dispatch!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:411:in `call!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `instance_eval'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `invoke'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `catch'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `invoke'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:411:in `call!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:399:in `call'
> ././lib/sinatra/rack_driver_select.rb:45:in `call'
> ././lib/sinatra/rack_matrix_params.rb:80:in `call'
> ././lib/sinatra/rack_runtime.rb:36:in `call'
> ././lib/sinatra/rack_etag.rb:42:in `call'
> /usr/lib/ruby/gems/1.8/gems/rack-accept-0.4.3/lib/rack/accept/context.rb:22:in `call'
> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/methodoverride.rb:24:in `call'
> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/commonlogger.rb:18:in `call'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:979:in `call'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:1005:in `synchronize'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:979:in `call'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/stats.rb:24:in `call'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/stats.rb:33:in `log'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/stats.rb:24:in `call'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/connection.rb:76:in `pre_process'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/connection.rb:74:in `catch'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/connection.rb:74:in `pre_process'
> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1060:in `call'
> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1060:in `spawn_threadpool'
> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1057:in `initialize'
> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1057:in `new'
> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1057:in `spawn_threadpool'
> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1049:in `defer'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/connection.rb:54:in `process'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/connection.rb:42:in `receive_data'
> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run_machine'
> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/backends/base.rb:57:in `start'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/server.rb:156:in `start'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/controllers/controller.rb:80:in `start'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/runner.rb:177:in `send'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/runner.rb:177:in `run_command'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/runner.rb:143:in `run!'
> /usr/share/deltacloud-core/bin/deltacloudd:124
> /usr/bin/deltacloudd:5:in `load'
> /usr/bin/deltacloudd:5
> 127.0.0.1 - - [31/May/2011 20:54:32] "GET /api/instances HTTP/1.1" 500 5628 9.4123
> 127.0.0.1 - - [31/May/2011 20:54:33] "GET /api/instances HTTP/1.1" 200 52194 9.9548
> Deltacloud::BackendError - RequestLimitExceeded: Request limit exceeded.
>
>
> --
> This message is automatically generated by JIRA.
> For more information on JIRA, see: http://www.atlassian.com/software/jira
------------------------------------------------------
Michal Fojtik, mfojtik@redhat.com
Deltacloud API: http://deltacloud.org
[jira] [Commented] (DTACLOUD-47) Request limit exceeded." response
from ec2, get tracebacks in deltacloud log
Posted by "Shveta Sachdeva (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DTACLOUD-47?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071605#comment-13071605 ]
Shveta Sachdeva commented on DTACLOUD-47:
-----------------------------------------
IDeltacloud::ExceptionHandler::ProviderError - InstanceLimitExceeded: 20 instance(s) are already running.
REQUEST=ec2.us-east-1.amazonaws.com:443/?AWSAccessKeyId=AKIAI2KPFDYVZKSRTJMQ&Action=RunInstances&AddressingType=public&ImageId=ami-d58147bc&InstanceType=c1.xlarge&KeyName=ssachdev%40redhat.com_Ec2_east_1311747031_key_70091981895960&MaxCount=1&MinCount=1&Placement.AvailabilityZone=us-east-1a&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2011-07-27T08%3A54%3A33.000Z&Version=2010-08-31&Signature=lf16hE28aWGQk3iX%2F6RsjvQeHlIg8mTVxrve2epNbpI%3D
REQUEST ID=cf9b9e48-80b3-472b-8670-bafbba59cee7 :
/usr/lib/ruby/gems/1.8/gems/aws-2.5.5/lib/ses/../awsbase/awsbase.rb:572:in `request_info_impl'
/usr/lib/ruby/gems/1.8/gems/aws-2.5.5/lib/ec2/ec2.rb:177:in `request_info'
/usr/lib/ruby/gems/1.8/gems/aws-2.5.5/lib/ec2/ec2.rb:577:in `launch_instances'
././lib/deltacloud/drivers/ec2/ec2_driver.rb:204:in `create_instance'
././lib/deltacloud/base_driver/exceptions.rb:111:in `call'
././lib/deltacloud/base_driver/exceptions.rb:111:in `safely'
././lib/deltacloud/drivers/ec2/ec2_driver.rb:203:in `create_instance'
././server.rb:397
././lib/sinatra/rabbit.rb:113:in `instance_eval'
././lib/sinatra/rabbit.rb:113:in `POST /api/instances'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:863:in `call'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:863:in `route'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:521:in `instance_eval'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:521:in `route_eval'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:500:in `route!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:497:in `catch'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:497:in `route!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:476:in `each'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:476:in `route!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:601:in `dispatch!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:411:in `call!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `instance_eval'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `invoke'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `catch'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `invoke'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:411:in `call!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:399:in `call'
././lib/sinatra/rack_driver_select.rb:45:in `call'
././lib/sinatra/rack_matrix_params.rb:85:in `call'
././lib/sinatra/rack_runtime.rb:36:in `call'
././lib/sinatra/rack_etag.rb:42:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-accept-0.4.3/lib/rack/accept/context.rb:22:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/methodoverride.rb:24:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/commonlogger.rb:18:in `call'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:979:in `call'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:1005:in `synchronize'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:979:in `call'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/connection.rb:76:in `pre_process'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/connection.rb:74:in `catch'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/connection.rb:74:in `pre_process'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1060:in `call'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1060:in `spawn_threadpool'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1057:in `initialize'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1057:in `new'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1057:in `spawn_threadpool'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1049:in `defer'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/connection.rb:54:in `process'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/connection.rb:42:in `receive_data'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run_machine'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/backends/base.rb:57:in `start'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/server.rb:156:in `start'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/controllers/controller.rb:80:in `start'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/runner.rb:177:in `send'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/runner.rb:177:in `run_command'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/runner.rb:143:in `run!'
/usr/share/deltacloud-core/bin/deltacloudd:126
/usr/bin/deltacloudd:5:in `load'
/usr/bin/deltacloudd:5
NFO -- : Launching instance of image ami-d58147bc for AKIAI2KPFDYVZKSRTJMQ, key: ssachdev@redhat.com_Ec2_east_1311747031_key_70091981895960, groups:
W, [2011-07-27T14:24:34.254207 #28723] WARN -- : ##### Aws::Ec2 returned an error: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InstanceLimitExceeded</Code><Message>20 instance(s) are already running.</Message></Error></Errors><RequestID>cf9b9e48-80b3-472b-8670-bafbba59cee7</RequestID></Response> #####
W, [2011-07-27T14:24:34.254287 #28723] WARN -- : ##### Aws::Ec2 request: ec2.us-east-1.amazonaws.com:443/?AWSAccessKeyId=AKIAI2KPFDYVZKSRTJMQ&Action=RunInstances&AddressingType=public&ImageId=ami-d58147bc&InstanceType=c1.xlarge&KeyName=ssachdev%40redhat.com_Ec2_east_1311747031_key_70091981895960&MaxCount=1&MinCount=1&Placement.AvailabilityZone=us-east-1a&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2011-07-27T08%3A54%3A33.000Z&Version=2010-08-31&Signature=lf16hE28aWGQk3iX%2F6RsjvQeHlIg8mTVxrve2epNbpI%3D ####
!! Unexpected error while processing request: undefined method `details' for #<Deltacloud::ExceptionHandler::ProviderError:0x7f983849dc48>
undefined method `details' for #<Deltacloud::ExceptionHandler::ProviderError:0x7f983849dc48>
././views/errors/502.xml.haml:7:in `__tilt_70145880646160'
././views/errors/502.xml.haml:-6:in `instance_eval'
././views/errors/502.xml.haml:-6:in `__tilt_70145880646160'
/usr/lib/ruby/gems/1.8/gems/tilt-1.3.2/lib/tilt/template.rb:140:in `call'
/usr/lib/ruby/gems/1.8/gems/tilt-1.3.2/lib/tilt/template.rb:140:in `cached_evaluate'
/usr/lib/ruby/gems/1.8/gems/tilt-1.3.2/lib/tilt/template.rb:127:in `evaluate'
/usr/lib/ruby/gems/1.8/gems/tilt-1.3.2/lib/tilt/haml.rb:24:in `evaluate'
/usr/lib/ruby/gems/1.8/gems/tilt-1.3.2/lib/tilt/template.rb:76:in `render'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:343:in `render_without_format'
././lib/sinatra/respond_to.rb:129:in `render'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:311:in `haml'
././lib/deltacloud/helpers/application_helper.rb:112:in `report_error'
././lib/sinatra/respond_to.rb:242:in `call'
././lib/sinatra/respond_to.rb:242:in `respond_to'
././lib/deltacloud/helpers/application_helper.rb:111:in `report_error'
././server.rb:61
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:641:in `instance_eval'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:641:in `error_block!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:636:in `each'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:636:in `error_block!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:630:in `handle_exception!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:605:in `dispatch!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:411:in `call!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `instance_eval'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `invoke'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `catch'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `invoke'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:411:in `call!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:399:in `call'
././lib/sinatra/rack_driver_select.rb:45:in `call'
././lib/sinatra/rack_matrix_params.rb:85:in `call'
././lib/sinatra/rack_runtime.rb:36:in `call'
././lib/sinatra/rack_etag.rb:42:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-accept-0.4.3/lib/rack/accept/context.rb:22:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/methodoverride.rb:24:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/commonlogger.rb:18:in `call'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:979:in `call'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:1005:in `synchronize'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:979:in `call'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/connection.rb:76:in `pre_process'
/usr/lib/ruby/gem127.0.0.1 - - [27/Jul/2011 14:24:34] "GET /api HTTP/1.1" 200 1439 0.2516
127.0.0.1 - - [27/Jul/2011 14:24:34] "GET /api/instances HTTP/1.1" 200 28789 1.4101
127.0.0.1 - - [27/Jul/2011 14:24:35] "GET /api/instances HTTP/1.1" 200 28789 1.6196
127.0.0.1 - - [27/Jul/2011 14:24:35] "GET /api HTTP/1.1" 200 1439 0.0565
127.0.0.1 - - [27/Jul/2011 14:24:36] "GET /api/instances HTTP/1.1" 200 28789 1.9271
127.0.0.1 - - [27/Jul/2011 14:24:36] "GET /api HTTP/1.1" 200 1439 0.1816
127.0.0.1 - - [27/Jul/2011 14:24:36] "GET /api HTTP/1.1" 200 1439 0.1850
127.0.0.1 - - [27/Jul/2011 14:24:38] "GET /api/instances HTTP/1.1" 200 28789 2.1307
127.0.0.1 - - [27/Jul/2011 14:24:39] "GET /api/instances HTTP/1.1" 200 28789 3.9238
127.0.0.1 - - [27/Jul/2011 14:24:40] "GET /api/instances HTTP/1.1" 200 28789 3.6063
127.0.0.1 - - [27/Jul/2011 14:24:40] "GET /api HTTP/1.1" 200 1439 0.0291
127.0.0.1 - - [27/Jul/2011 14:24:41] "GET /api HTTP/1.1" 200 1439 0.0219
127.0.0.1 - - [27/Jul/2011 14:24:42] "GET /api/instances HTTP/1.1" 200 28789 1.7214
127.0.0.1 - - [27/Jul/2011 14:24:43] "GET /api/instances HTTP/1.1" 200 28789 3.4474
127.0.0.1 - - [27/Jul/2011 14:24:43] "GET /api HTTP/1.1" 200 1439 0.0270
127.0.0.1 - - [27/Jul/2011 14:24:43] "GET /api HTTP/1.1" 200 1439 0.0285
127.0.0.1 - - [27/Jul/2011 14:24:45] "GET /api HTTP/1.1" 200 1439 0.0520
127.0.0.1 - - [27/Jul/2011 14:24:45] "GET /api HTTP/1.1" 200 1439 0.0553
127.0.0.1 - - [27/Jul/2011 14:24:45] "GET /api/instances HTTP/1.1" 200 28789 1.6354
127.0.0.1 - - [27/Jul/2011 14:24:45] "GET /api/instances HTTP/1.1" 200 28789 1.7878
127.0.0.1 - - [27/Jul/2011 14:24:47] "GET /api/instances HTTP/1.1" 200 28789 2.1723
127.0.0.1 - - [27/Jul/2011 14:24:47] "GET /api/instances HTTP/1.1" 200 28789 2.7255
127.0.0.1 - - [27/Jul/2011 14:24:49] "GET /api HTTP/1.1" 200 1439 0.0450
127.0.0.1 - - [27/Jul/2011 14:24:49] "GET /api HTTP/1.1" 200 1439 0.0380
127.0.0.1 - - [27/Jul/2011 14:24:50] "GET /api/instances HTTP/1.1" 200 28789 1.5791
127.0.0.1 - - [27/Jul/2011 14:24:51] "GET /api/instances HTTP/1.1" 200 28789 1.8838
> Request limit exceeded." response from ec2, get tracebacks in deltacloud log
> ----------------------------------------------------------------------------
>
> Key: DTACLOUD-47
> URL: https://issues.apache.org/jira/browse/DTACLOUD-47
> Project: DeltaCloud
> Issue Type: Bug
> Components: Client (Ruby)
> Environment: ec2 + aeolus
> Reporter: wes hayutin
> Assignee: David Lutterkort
>
> Under heavy use, starting 15-20 instances serially its possible to get a response back from ec2 that the "request limit has been reached". Once a load is created, and ec2 can not handle more requests I see a lot of tracebacks in the ec2 log.
> 127.0.0.1 - - [31/May/2011 20:54:32] "GET /api/instances HTTP/1.1" 500 5628 9.4191
> 127.0.0.1 - - [31/May/2011 20:54:32] "GET /api/instances HTTP/1.1" 200 52194 9.8775
> Deltacloud::BackendError - RequestLimitExceeded: Request limit exceeded.
> REQUEST=ec2.us-east-1.amazonaws.com:443/?AWSAccessKeyId=AKIAJ557V2EQ&Action=DeleteTags&ResourceId.1=i-fba01795&SignatureMethod=HmacSHA256&SignatureVersion=2&Tag.1.Key=name&Timestamp=2011-06-01T00%3A54%3A32.000Z&Version=2010-08-31&Signature=ZLFgXz2V9aIIyuV9FTHApP4vNnfo8NxFaf6OEhSCJss%3D
> REQUEST ID=55502bf8-4814-41a6-933b-fb02b6a21e20 :
> ././lib/deltacloud/base_driver/base_driver.rb:247:in `safely'
> ././lib/deltacloud/base_driver/base_driver.rb:245:in `each'
> ././lib/deltacloud/base_driver/base_driver.rb:245:in `safely'
> ././lib/deltacloud/drivers/ec2/ec2_driver.rb:174:in `instances'
> ././lib/deltacloud/helpers/application_helper.rb:80:in `send'
> ././lib/deltacloud/helpers/application_helper.rb:80:in `filter_all'
> /usr/lib/ruby/1.8/benchmark.rb:293:in `measure'
> ././lib/deltacloud/helpers/application_helper.rb:79:in `filter_all'
> ././server.rb:383
> ././lib/sinatra/rabbit.rb:104:in `instance_eval'
> ././lib/sinatra/rabbit.rb:104:in `GET /api/instances'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:863:in `call'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:863:in `route'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:521:in `instance_eval'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:521:in `route_eval'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:500:in `route!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:497:in `catch'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:497:in `route!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:476:in `each'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:476:in `route!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:601:in `dispatch!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:411:in `call!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `instance_eval'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `invoke'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `catch'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `invoke'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:411:in `call!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:399:in `call'
> ././lib/sinatra/rack_driver_select.rb:45:in `call'
> ././lib/sinatra/rack_matrix_params.rb:80:in `call'
> ././lib/sinatra/rack_runtime.rb:36:in `call'
> ././lib/sinatra/rack_etag.rb:42:in `call'
> /usr/lib/ruby/gems/1.8/gems/rack-accept-0.4.3/lib/rack/accept/context.rb:22:in `call'
> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/methodoverride.rb:24:in `call'
> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/commonlogger.rb:18:in `call'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:979:in `call'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:1005:in `synchronize'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:979:in `call'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/stats.rb:24:in `call'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/stats.rb:33:in `log'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/stats.rb:24:in `call'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/connection.rb:76:in `pre_process'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/connection.rb:74:in `catch'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/connection.rb:74:in `pre_process'
> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1060:in `call'
> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1060:in `spawn_threadpool'
> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1057:in `initialize'
> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1057:in `new'
> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1057:in `spawn_threadpool'
> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1049:in `defer'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/connection.rb:54:in `process'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/connection.rb:42:in `receive_data'
> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run_machine'
> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/backends/base.rb:57:in `start'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/server.rb:156:in `start'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/controllers/controller.rb:80:in `start'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/runner.rb:177:in `send'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/runner.rb:177:in `run_command'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/runner.rb:143:in `run!'
> /usr/share/deltacloud-core/bin/deltacloudd:124
> /usr/bin/deltacloudd:5:in `load'
> /usr/bin/deltacloudd:5
> 127.0.0.1 - - [31/May/2011 20:54:32] "GET /api/instances HTTP/1.1" 500 5628 9.4123
> 127.0.0.1 - - [31/May/2011 20:54:33] "GET /api/instances HTTP/1.1" 200 52194 9.9548
> Deltacloud::BackendError - RequestLimitExceeded: Request limit exceeded.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira