[jira] [Updated] (WHIRR-455) Whirr cannot create large ec2 clusters -- too chatty -- RequestLimitExceeded

["Bob Cotton (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/WHIRR-455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bob Cotton updated WHIRR-455:
-----------------------------

    Attachment: whirr.log.too-many-requests.gz
    
> Whirr cannot create large ec2 clusters -- too chatty -- RequestLimitExceeded
> ----------------------------------------------------------------------------
>
>                 Key: WHIRR-455
>                 URL: https://issues.apache.org/jira/browse/WHIRR-455
>             Project: Whirr
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 0.7.0
>         Environment: bash-3.2$ uname -a
> Darwin bcotton-mbp.f4tech.com 11.2.0 Darwin Kernel Version 11.2.0: Tue Aug  9 20:54:00 PDT 2011; root:xnu-1699.24.8~1/RELEASE_X86_64 x86_64
> bash-3.2$ java -version
> java version "1.6.0_29"
> Java(TM) SE Runtime Environment (build 1.6.0_29-b11-402-11M3527)
> Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed mode)
>            Reporter: Bob Cotton
>         Attachments: whirr.log.too-many-requests.gz
>
>
> Trying to use whirr to create clusters of 50+ nodes on ec2 using spot instances.
> It will eventually fail trying to poll the security groups for the pending instances, and eventually ec2 will block the requests due to the number of requests.
> The spot bids will be created, and instances will be started in them, but whirr can't see them due to being locked out.
> I will attach a log file, with jclouds ec2 logging turned up. This may be a jclouds issue, but I figured I'd start here.
> configuration file:
> # Change the cluster name here
> whirr.cluster-name=hadoop
> # Change the number of machines in the cluster here
> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,50 hadoop-datanode+hadoop-tasktracker
> # Uncomment out these lines to run CDH
> whirr.hadoop.install-function=install_cdh_hadoop
> whirr.hadoop.configure-function=configure_cdh_hadoop
> # For EC2 set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
> whirr.provider=aws-ec2
> whirr.identity=
> whirr.credential=
> # The size of the instance to use. See http://aws.amazon.com/ec2/instance-types/
> whirr.hardware-id=c1.xlarge
> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/
> whirr.image-id=us-east-1/ami-da0cf8b3
> # If you choose a different location, make sure whirr.image-id is updated too
> whirr.location-id=us-east-1a
> # By default use the user system SSH keys. Override them here.
> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa_hadoop
> whirr.public-key-file=${whirr.private-key-file}.pub
> # You can also specify the spot instance price
> # http://aws.amazon.com/ec2/spot-instances/
> whirr.aws-ec2-spot-price=0.68

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira