You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@whirr.apache.org by Sean Zhang <zs...@gmail.com> on 2012/03/21 19:33:46 UTC

using cluster computing instances

Hi,

I am having problem using cluster instances -- cc1.4xlarge.
I have tried several different hvm images. Whirr always give the same error
as below. Does cluster instances supported?

Caused by: java.util.NoSuchElementException: no hardware profiles support
images matching params: [biggest=false, fastest=false, imageName=null,
imageDescription=ubuntu-images/, imageId=null, imagePredicate=null,
imageVersion=null, location=[id=us-east-1, scope=REGION,
description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA], metadata={}],
minCores=0.0, minRam=0, osFamily=ubuntu, osName=null, osDescription=null,
osVersion=10.04, osArch=null, os64Bit=null, hardwareId=cc1.4xlarge]
at
org.jclouds.compute.domain.internal.TemplateBuilderImpl.resolveSize(TemplateBuilderImpl.java:614)
at
org.jclouds.compute.domain.internal.TemplateBuilderImpl.build(TemplateBuilderImpl.java:573)
at
org.apache.whirr.compute.BootstrapTemplate.build(BootstrapTemplate.java:75)
at
org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:100)
at
org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:107)
at
org.apache.whirr.ClusterController.launchCluster(ClusterController.java:106)
... 3 more

Thanks,
Sean

Re: using cluster computing instances

Posted by Andrei Savu <sa...@gmail.com>.
Unfortunately there is no formal documentation that explains the Whirr core
but let me try to give you a quick overview:

* the deployment of a cluster is a multi-phase process
* the architecture is described using a cluster specification stored as a
.properties file. This file has both role assignments and configuration
properties for each service as needed

* before 0.8.0 we only had 3 phases: bootstrap (install), configure, destroy
* in 0.8.0 we want to improve the cluster management capabilities by adding
start, stop, cleanup (uninstall)

* we are making extensive use of jclouds for machine provisioning and
script execution
* we are using the compute and the blobstore APIs

* in the bootstrap phase we make machine provisioning requests and we
execute the bootstrap / service install scripts
* the bootstrap phase is resilient to machine failure
* this is the phase you need to change to be able to make use of cluster
compute instances and placement groups
* see
https://github.com/apache/whirr/blob/trunk/core/src/main/java/org/apache/whirr/compute/BootstrapTemplate.java

* in the configure phase we usually generate configuration files based on
machine IPs and hostnames
* we ship those files and a set of scripts in parallel for execution on the
cluster machines

I am sure you will have more questions as you go through the code and we
are available to answer them either on the email list or on IRC at #whirr
(freenode).

To understand the structure of a service you can check the following
presentations:
https://cwiki.apache.org/confluence/display/WHIRR/Presentations

... and the implementation for a simple one (e.g. elasticsearch or
zookeeper).

-- Andrei Savu / andreisavu.ro

On Wed, Mar 21, 2012 at 9:56 PM, Sean Zhang <zs...@gmail.com> wrote:

> Is there some high level design document for the project? I would like to
> read something before start digging into the code. anything you can suggest?
>
> Thanks,
> Sean
>
>
> On Wed, Mar 21, 2012 at 3:53 PM, Andrei Savu <sa...@gmail.com>wrote:
>
>>
>> On Wed, Mar 21, 2012 at 8:49 PM, Sean Zhang <zs...@gmail.com> wrote:
>>
>>> could it be something related to:
>>> https://issues.apache.org/jira/browse/WHIRR-63
>>>
>>
>> I think you may be right but I'm not sure - I've never tried to start a
>> cluster compute instance before.
>>
>> You are more than welcome to lead the effort of making #63 a reality! :)
>>
>
>

Re: using cluster computing instances

Posted by Sean Zhang <zs...@gmail.com>.
Is there some high level design document for the project? I would like to
read something before start digging into the code. anything you can suggest?

Thanks,
Sean

On Wed, Mar 21, 2012 at 3:53 PM, Andrei Savu <sa...@gmail.com> wrote:

>
> On Wed, Mar 21, 2012 at 8:49 PM, Sean Zhang <zs...@gmail.com> wrote:
>
>> could it be something related to:
>> https://issues.apache.org/jira/browse/WHIRR-63
>>
>
> I think you may be right but I'm not sure - I've never tried to start a
> cluster compute instance before.
>
> You are more than welcome to lead the effort of making #63 a reality! :)
>

Re: using cluster computing instances

Posted by Andrei Savu <sa...@gmail.com>.
On Wed, Mar 21, 2012 at 8:49 PM, Sean Zhang <zs...@gmail.com> wrote:

> could it be something related to:
> https://issues.apache.org/jira/browse/WHIRR-63
>

I think you may be right but I'm not sure - I've never tried to start a
cluster compute instance before.

You are more than welcome to lead the effort of making #63 a reality! :)

Re: using cluster computing instances

Posted by Sean Zhang <zs...@gmail.com>.
cc1.4xlarge requires hvm images. I have tried both suse, amazon and
ubuntu10.04 and ended up with the same error.

could it be something related to:
https://issues.apache.org/jira/browse/WHIRR-63

On Wed, Mar 21, 2012 at 2:46 PM, Andrei Savu <sa...@gmail.com> wrote:

> My quick guess is that there is no ubuntu 10.04 AMI that supports HVM /
> cluster compute instances but I may be wrong.
>
> Do you need that instance type? I think your best bet is to use amazon
> linux: http://aws.amazon.com/amazon-linux-ami/
>
>
> On Wed, Mar 21, 2012 at 8:33 PM, Sean Zhang <zs...@gmail.com> wrote:
>
>> Hi,
>>
>> I am having problem using cluster instances -- cc1.4xlarge.
>> I have tried several different hvm images. Whirr always give the same
>> error as below. Does cluster instances supported?
>>
>> Caused by: java.util.NoSuchElementException: no hardware profiles support
>> images matching params: [biggest=false, fastest=false, imageName=null,
>> imageDescription=ubuntu-images/, imageId=null, imagePredicate=null,
>> imageVersion=null, location=[id=us-east-1, scope=REGION,
>> description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA], metadata={}],
>> minCores=0.0, minRam=0, osFamily=ubuntu, osName=null, osDescription=null,
>> osVersion=10.04, osArch=null, os64Bit=null, hardwareId=cc1.4xlarge]
>>  at
>> org.jclouds.compute.domain.internal.TemplateBuilderImpl.resolveSize(TemplateBuilderImpl.java:614)
>> at
>> org.jclouds.compute.domain.internal.TemplateBuilderImpl.build(TemplateBuilderImpl.java:573)
>>  at
>> org.apache.whirr.compute.BootstrapTemplate.build(BootstrapTemplate.java:75)
>> at
>> org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:100)
>>  at
>> org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:107)
>> at
>> org.apache.whirr.ClusterController.launchCluster(ClusterController.java:106)
>>  ... 3 more
>>
>> Thanks,
>> Sean
>>
>
>

Re: using cluster computing instances

Posted by Andrei Savu <sa...@gmail.com>.
My quick guess is that there is no ubuntu 10.04 AMI that supports HVM /
cluster compute instances but I may be wrong.

Do you need that instance type? I think your best bet is to use amazon
linux: http://aws.amazon.com/amazon-linux-ami/

On Wed, Mar 21, 2012 at 8:33 PM, Sean Zhang <zs...@gmail.com> wrote:

> Hi,
>
> I am having problem using cluster instances -- cc1.4xlarge.
> I have tried several different hvm images. Whirr always give the same
> error as below. Does cluster instances supported?
>
> Caused by: java.util.NoSuchElementException: no hardware profiles support
> images matching params: [biggest=false, fastest=false, imageName=null,
> imageDescription=ubuntu-images/, imageId=null, imagePredicate=null,
> imageVersion=null, location=[id=us-east-1, scope=REGION,
> description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA], metadata={}],
> minCores=0.0, minRam=0, osFamily=ubuntu, osName=null, osDescription=null,
> osVersion=10.04, osArch=null, os64Bit=null, hardwareId=cc1.4xlarge]
>  at
> org.jclouds.compute.domain.internal.TemplateBuilderImpl.resolveSize(TemplateBuilderImpl.java:614)
> at
> org.jclouds.compute.domain.internal.TemplateBuilderImpl.build(TemplateBuilderImpl.java:573)
>  at
> org.apache.whirr.compute.BootstrapTemplate.build(BootstrapTemplate.java:75)
> at
> org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:100)
>  at
> org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:107)
> at
> org.apache.whirr.ClusterController.launchCluster(ClusterController.java:106)
>  ... 3 more
>
> Thanks,
> Sean
>