You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@whirr.apache.org by "Bob Cotton (Created) (JIRA)" <ji...@apache.org> on 2011/12/20 21:27:30 UTC
[jira] [Created] (WHIRR-455) Whirr cannot create large ec2 clusters
-- too chatty -- RequestLimitExceeded
Whirr cannot create large ec2 clusters -- too chatty -- RequestLimitExceeded
----------------------------------------------------------------------------
Key: WHIRR-455
URL: https://issues.apache.org/jira/browse/WHIRR-455
Project: Whirr
Issue Type: Bug
Components: core
Affects Versions: 0.7.0
Environment: bash-3.2$ uname -a
Darwin bcotton-mbp.f4tech.com 11.2.0 Darwin Kernel Version 11.2.0: Tue Aug 9 20:54:00 PDT 2011; root:xnu-1699.24.8~1/RELEASE_X86_64 x86_64
bash-3.2$ java -version
java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11-402-11M3527)
Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed mode)
Reporter: Bob Cotton
Trying to use whirr to create clusters of 50+ nodes on ec2 using spot instances.
It will eventually fail trying to poll the security groups for the pending instances, and eventually ec2 will block the requests due to the number of requests.
The spot bids will be created, and instances will be started in them, but whirr can't see them due to being locked out.
I will attach a log file, with jclouds ec2 logging turned up. This may be a jclouds issue, but I figured I'd start here.
configuration file:
# Change the cluster name here
whirr.cluster-name=hadoop
# Change the number of machines in the cluster here
whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,50 hadoop-datanode+hadoop-tasktracker
# Uncomment out these lines to run CDH
whirr.hadoop.install-function=install_cdh_hadoop
whirr.hadoop.configure-function=configure_cdh_hadoop
# For EC2 set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
whirr.provider=aws-ec2
whirr.identity=
whirr.credential=
# The size of the instance to use. See http://aws.amazon.com/ec2/instance-types/
whirr.hardware-id=c1.xlarge
# Ubuntu 10.04 LTS Lucid. See http://alestic.com/
whirr.image-id=us-east-1/ami-da0cf8b3
# If you choose a different location, make sure whirr.image-id is updated too
whirr.location-id=us-east-1a
# By default use the user system SSH keys. Override them here.
whirr.private-key-file=${sys:user.home}/.ssh/id_rsa_hadoop
whirr.public-key-file=${whirr.private-key-file}.pub
# You can also specify the spot instance price
# http://aws.amazon.com/ec2/spot-instances/
whirr.aws-ec2-spot-price=0.68
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (WHIRR-455) Whirr cannot create large ec2
clusters -- too chatty -- RequestLimitExceeded
Posted by "Adrian Cole (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173486#comment-13173486 ]
Adrian Cole commented on WHIRR-455:
-----------------------------------
there are places that we could reduce chattiness, a noble goal. mind popping an issue here? http://code.google.com/p/jclouds/issues/entry
> Whirr cannot create large ec2 clusters -- too chatty -- RequestLimitExceeded
> ----------------------------------------------------------------------------
>
> Key: WHIRR-455
> URL: https://issues.apache.org/jira/browse/WHIRR-455
> Project: Whirr
> Issue Type: Bug
> Components: core
> Affects Versions: 0.7.0
> Environment: bash-3.2$ uname -a
> Darwin bcotton-mbp.f4tech.com 11.2.0 Darwin Kernel Version 11.2.0: Tue Aug 9 20:54:00 PDT 2011; root:xnu-1699.24.8~1/RELEASE_X86_64 x86_64
> bash-3.2$ java -version
> java version "1.6.0_29"
> Java(TM) SE Runtime Environment (build 1.6.0_29-b11-402-11M3527)
> Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed mode)
> Reporter: Bob Cotton
> Attachments: whirr.log.too-many-requests.gz
>
>
> Trying to use whirr to create clusters of 50+ nodes on ec2 using spot instances.
> It will eventually fail trying to poll the security groups for the pending instances, and eventually ec2 will block the requests due to the number of requests.
> The spot bids will be created, and instances will be started in them, but whirr can't see them due to being locked out.
> I will attach a log file, with jclouds ec2 logging turned up. This may be a jclouds issue, but I figured I'd start here.
> configuration file:
> # Change the cluster name here
> whirr.cluster-name=hadoop
> # Change the number of machines in the cluster here
> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,50 hadoop-datanode+hadoop-tasktracker
> # Uncomment out these lines to run CDH
> whirr.hadoop.install-function=install_cdh_hadoop
> whirr.hadoop.configure-function=configure_cdh_hadoop
> # For EC2 set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
> whirr.provider=aws-ec2
> whirr.identity=
> whirr.credential=
> # The size of the instance to use. See http://aws.amazon.com/ec2/instance-types/
> whirr.hardware-id=c1.xlarge
> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/
> whirr.image-id=us-east-1/ami-da0cf8b3
> # If you choose a different location, make sure whirr.image-id is updated too
> whirr.location-id=us-east-1a
> # By default use the user system SSH keys. Override them here.
> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa_hadoop
> whirr.public-key-file=${whirr.private-key-file}.pub
> # You can also specify the spot instance price
> # http://aws.amazon.com/ec2/spot-instances/
> whirr.aws-ec2-spot-price=0.68
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-455) Whirr cannot create large ec2 clusters
-- too chatty -- RequestLimitExceeded
Posted by "Bob Cotton (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bob Cotton updated WHIRR-455:
-----------------------------
Attachment: whirr.log.too-many-requests.gz
> Whirr cannot create large ec2 clusters -- too chatty -- RequestLimitExceeded
> ----------------------------------------------------------------------------
>
> Key: WHIRR-455
> URL: https://issues.apache.org/jira/browse/WHIRR-455
> Project: Whirr
> Issue Type: Bug
> Components: core
> Affects Versions: 0.7.0
> Environment: bash-3.2$ uname -a
> Darwin bcotton-mbp.f4tech.com 11.2.0 Darwin Kernel Version 11.2.0: Tue Aug 9 20:54:00 PDT 2011; root:xnu-1699.24.8~1/RELEASE_X86_64 x86_64
> bash-3.2$ java -version
> java version "1.6.0_29"
> Java(TM) SE Runtime Environment (build 1.6.0_29-b11-402-11M3527)
> Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed mode)
> Reporter: Bob Cotton
> Attachments: whirr.log.too-many-requests.gz
>
>
> Trying to use whirr to create clusters of 50+ nodes on ec2 using spot instances.
> It will eventually fail trying to poll the security groups for the pending instances, and eventually ec2 will block the requests due to the number of requests.
> The spot bids will be created, and instances will be started in them, but whirr can't see them due to being locked out.
> I will attach a log file, with jclouds ec2 logging turned up. This may be a jclouds issue, but I figured I'd start here.
> configuration file:
> # Change the cluster name here
> whirr.cluster-name=hadoop
> # Change the number of machines in the cluster here
> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,50 hadoop-datanode+hadoop-tasktracker
> # Uncomment out these lines to run CDH
> whirr.hadoop.install-function=install_cdh_hadoop
> whirr.hadoop.configure-function=configure_cdh_hadoop
> # For EC2 set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
> whirr.provider=aws-ec2
> whirr.identity=
> whirr.credential=
> # The size of the instance to use. See http://aws.amazon.com/ec2/instance-types/
> whirr.hardware-id=c1.xlarge
> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/
> whirr.image-id=us-east-1/ami-da0cf8b3
> # If you choose a different location, make sure whirr.image-id is updated too
> whirr.location-id=us-east-1a
> # By default use the user system SSH keys. Override them here.
> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa_hadoop
> whirr.public-key-file=${whirr.private-key-file}.pub
> # You can also specify the spot instance price
> # http://aws.amazon.com/ec2/spot-instances/
> whirr.aws-ec2-spot-price=0.68
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (WHIRR-455) Whirr cannot create large ec2
clusters -- too chatty -- RequestLimitExceeded
Posted by "Andrei Savu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173492#comment-13173492 ]
Andrei Savu commented on WHIRR-455:
-----------------------------------
Thanks Bob for opening this JIRA. We are trying to provide an workaround for this in 0.8.0 by implementing the ability to resize existing clusters. This should allow you to go from 5 to 50 by adding 5 new machines on each iteration. Sounds reasonable? Do you think this approach could work for you?
Also see this page http://www.jclouds.org/documentation/reference/pool-design for some of the things we are working on in jclouds.
> Whirr cannot create large ec2 clusters -- too chatty -- RequestLimitExceeded
> ----------------------------------------------------------------------------
>
> Key: WHIRR-455
> URL: https://issues.apache.org/jira/browse/WHIRR-455
> Project: Whirr
> Issue Type: Bug
> Components: core
> Affects Versions: 0.7.0
> Environment: bash-3.2$ uname -a
> Darwin bcotton-mbp.f4tech.com 11.2.0 Darwin Kernel Version 11.2.0: Tue Aug 9 20:54:00 PDT 2011; root:xnu-1699.24.8~1/RELEASE_X86_64 x86_64
> bash-3.2$ java -version
> java version "1.6.0_29"
> Java(TM) SE Runtime Environment (build 1.6.0_29-b11-402-11M3527)
> Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed mode)
> Reporter: Bob Cotton
> Attachments: whirr.log.too-many-requests.gz
>
>
> Trying to use whirr to create clusters of 50+ nodes on ec2 using spot instances.
> It will eventually fail trying to poll the security groups for the pending instances, and eventually ec2 will block the requests due to the number of requests.
> The spot bids will be created, and instances will be started in them, but whirr can't see them due to being locked out.
> I will attach a log file, with jclouds ec2 logging turned up. This may be a jclouds issue, but I figured I'd start here.
> configuration file:
> # Change the cluster name here
> whirr.cluster-name=hadoop
> # Change the number of machines in the cluster here
> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,50 hadoop-datanode+hadoop-tasktracker
> # Uncomment out these lines to run CDH
> whirr.hadoop.install-function=install_cdh_hadoop
> whirr.hadoop.configure-function=configure_cdh_hadoop
> # For EC2 set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
> whirr.provider=aws-ec2
> whirr.identity=
> whirr.credential=
> # The size of the instance to use. See http://aws.amazon.com/ec2/instance-types/
> whirr.hardware-id=c1.xlarge
> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/
> whirr.image-id=us-east-1/ami-da0cf8b3
> # If you choose a different location, make sure whirr.image-id is updated too
> whirr.location-id=us-east-1a
> # By default use the user system SSH keys. Override them here.
> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa_hadoop
> whirr.public-key-file=${whirr.private-key-file}.pub
> # You can also specify the spot instance price
> # http://aws.amazon.com/ec2/spot-instances/
> whirr.aws-ec2-spot-price=0.68
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (WHIRR-455) Whirr cannot create large ec2
clusters -- too chatty -- RequestLimitExceeded
Posted by "Bob Cotton (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173499#comment-13173499 ]
Bob Cotton commented on WHIRR-455:
----------------------------------
It would at least be a start if I could add machines to the cluster at a later time.
I rather use something like whirr, because EMR has poor visibility.
Thanks
> Whirr cannot create large ec2 clusters -- too chatty -- RequestLimitExceeded
> ----------------------------------------------------------------------------
>
> Key: WHIRR-455
> URL: https://issues.apache.org/jira/browse/WHIRR-455
> Project: Whirr
> Issue Type: Bug
> Components: core
> Affects Versions: 0.7.0
> Environment: bash-3.2$ uname -a
> Darwin bcotton-mbp.f4tech.com 11.2.0 Darwin Kernel Version 11.2.0: Tue Aug 9 20:54:00 PDT 2011; root:xnu-1699.24.8~1/RELEASE_X86_64 x86_64
> bash-3.2$ java -version
> java version "1.6.0_29"
> Java(TM) SE Runtime Environment (build 1.6.0_29-b11-402-11M3527)
> Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed mode)
> Reporter: Bob Cotton
> Attachments: whirr.log.too-many-requests.gz
>
>
> Trying to use whirr to create clusters of 50+ nodes on ec2 using spot instances.
> It will eventually fail trying to poll the security groups for the pending instances, and eventually ec2 will block the requests due to the number of requests.
> The spot bids will be created, and instances will be started in them, but whirr can't see them due to being locked out.
> I will attach a log file, with jclouds ec2 logging turned up. This may be a jclouds issue, but I figured I'd start here.
> configuration file:
> {noformat}
> # Change the cluster name here
> whirr.cluster-name=hadoop
> # Change the number of machines in the cluster here
> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,50 hadoop-datanode+hadoop-tasktracker
> # Uncomment out these lines to run CDH
> whirr.hadoop.install-function=install_cdh_hadoop
> whirr.hadoop.configure-function=configure_cdh_hadoop
> # For EC2 set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
> whirr.provider=aws-ec2
> whirr.identity=
> whirr.credential=
> # The size of the instance to use. See http://aws.amazon.com/ec2/instance-types/
> whirr.hardware-id=c1.xlarge
> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/
> whirr.image-id=us-east-1/ami-da0cf8b3
> # If you choose a different location, make sure whirr.image-id is updated too
> whirr.location-id=us-east-1a
> # By default use the user system SSH keys. Override them here.
> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa_hadoop
> whirr.public-key-file=${whirr.private-key-file}.pub
> # You can also specify the spot instance price
> # http://aws.amazon.com/ec2/spot-instances/
> whirr.aws-ec2-spot-price=0.68
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-455) Whirr cannot create large ec2 clusters
-- too chatty -- RequestLimitExceeded
Posted by "Andrei Savu (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Savu updated WHIRR-455:
------------------------------
Description:
Trying to use whirr to create clusters of 50+ nodes on ec2 using spot instances.
It will eventually fail trying to poll the security groups for the pending instances, and eventually ec2 will block the requests due to the number of requests.
The spot bids will be created, and instances will be started in them, but whirr can't see them due to being locked out.
I will attach a log file, with jclouds ec2 logging turned up. This may be a jclouds issue, but I figured I'd start here.
configuration file:
{noformat}
# Change the cluster name here
whirr.cluster-name=hadoop
# Change the number of machines in the cluster here
whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,50 hadoop-datanode+hadoop-tasktracker
# Uncomment out these lines to run CDH
whirr.hadoop.install-function=install_cdh_hadoop
whirr.hadoop.configure-function=configure_cdh_hadoop
# For EC2 set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
whirr.provider=aws-ec2
whirr.identity=
whirr.credential=
# The size of the instance to use. See http://aws.amazon.com/ec2/instance-types/
whirr.hardware-id=c1.xlarge
# Ubuntu 10.04 LTS Lucid. See http://alestic.com/
whirr.image-id=us-east-1/ami-da0cf8b3
# If you choose a different location, make sure whirr.image-id is updated too
whirr.location-id=us-east-1a
# By default use the user system SSH keys. Override them here.
whirr.private-key-file=${sys:user.home}/.ssh/id_rsa_hadoop
whirr.public-key-file=${whirr.private-key-file}.pub
# You can also specify the spot instance price
# http://aws.amazon.com/ec2/spot-instances/
whirr.aws-ec2-spot-price=0.68
{noformat}
was:
Trying to use whirr to create clusters of 50+ nodes on ec2 using spot instances.
It will eventually fail trying to poll the security groups for the pending instances, and eventually ec2 will block the requests due to the number of requests.
The spot bids will be created, and instances will be started in them, but whirr can't see them due to being locked out.
I will attach a log file, with jclouds ec2 logging turned up. This may be a jclouds issue, but I figured I'd start here.
configuration file:
# Change the cluster name here
whirr.cluster-name=hadoop
# Change the number of machines in the cluster here
whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,50 hadoop-datanode+hadoop-tasktracker
# Uncomment out these lines to run CDH
whirr.hadoop.install-function=install_cdh_hadoop
whirr.hadoop.configure-function=configure_cdh_hadoop
# For EC2 set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
whirr.provider=aws-ec2
whirr.identity=
whirr.credential=
# The size of the instance to use. See http://aws.amazon.com/ec2/instance-types/
whirr.hardware-id=c1.xlarge
# Ubuntu 10.04 LTS Lucid. See http://alestic.com/
whirr.image-id=us-east-1/ami-da0cf8b3
# If you choose a different location, make sure whirr.image-id is updated too
whirr.location-id=us-east-1a
# By default use the user system SSH keys. Override them here.
whirr.private-key-file=${sys:user.home}/.ssh/id_rsa_hadoop
whirr.public-key-file=${whirr.private-key-file}.pub
# You can also specify the spot instance price
# http://aws.amazon.com/ec2/spot-instances/
whirr.aws-ec2-spot-price=0.68
> Whirr cannot create large ec2 clusters -- too chatty -- RequestLimitExceeded
> ----------------------------------------------------------------------------
>
> Key: WHIRR-455
> URL: https://issues.apache.org/jira/browse/WHIRR-455
> Project: Whirr
> Issue Type: Bug
> Components: core
> Affects Versions: 0.7.0
> Environment: bash-3.2$ uname -a
> Darwin bcotton-mbp.f4tech.com 11.2.0 Darwin Kernel Version 11.2.0: Tue Aug 9 20:54:00 PDT 2011; root:xnu-1699.24.8~1/RELEASE_X86_64 x86_64
> bash-3.2$ java -version
> java version "1.6.0_29"
> Java(TM) SE Runtime Environment (build 1.6.0_29-b11-402-11M3527)
> Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed mode)
> Reporter: Bob Cotton
> Attachments: whirr.log.too-many-requests.gz
>
>
> Trying to use whirr to create clusters of 50+ nodes on ec2 using spot instances.
> It will eventually fail trying to poll the security groups for the pending instances, and eventually ec2 will block the requests due to the number of requests.
> The spot bids will be created, and instances will be started in them, but whirr can't see them due to being locked out.
> I will attach a log file, with jclouds ec2 logging turned up. This may be a jclouds issue, but I figured I'd start here.
> configuration file:
> {noformat}
> # Change the cluster name here
> whirr.cluster-name=hadoop
> # Change the number of machines in the cluster here
> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,50 hadoop-datanode+hadoop-tasktracker
> # Uncomment out these lines to run CDH
> whirr.hadoop.install-function=install_cdh_hadoop
> whirr.hadoop.configure-function=configure_cdh_hadoop
> # For EC2 set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
> whirr.provider=aws-ec2
> whirr.identity=
> whirr.credential=
> # The size of the instance to use. See http://aws.amazon.com/ec2/instance-types/
> whirr.hardware-id=c1.xlarge
> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/
> whirr.image-id=us-east-1/ami-da0cf8b3
> # If you choose a different location, make sure whirr.image-id is updated too
> whirr.location-id=us-east-1a
> # By default use the user system SSH keys. Override them here.
> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa_hadoop
> whirr.public-key-file=${whirr.private-key-file}.pub
> # You can also specify the spot instance price
> # http://aws.amazon.com/ec2/spot-instances/
> whirr.aws-ec2-spot-price=0.68
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira