You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@whirr.apache.org by Andrew Bayer <an...@gmail.com> on 2013/03/15 21:20:26 UTC
Re: Whirr on HPCloud
Not to engage in thread necromancy, but did you ever manage to get security
groups on hp cloud working with whirr? =)
A.
On Sun, Aug 26, 2012 at 6:12 PM, Paul Baclace <pa...@gmail.com>wrote:
> > From: Adrian Cole
>
> > I suppose another way would be to use "openstack-nova-ec2" provider
> > and specify hp as the endpoint. This could possibly work, and then
> > you can use EC2 security group properties.
>
> I am able to get an HP AUTH token with curl and use it to list sec.
> groups, am not sure how to pass that to jclouds, so Whirr gets auth failure
> just looking up images. I tried setting whirr.credential to the HP AUTH
> token and also tried using the secret_key.
>
> Workaround would be to supply Whirr with a pre-exisiting security group to
> use, then I can modify the sec. group with a w.x.y.z/32, CIDR list after
> Whirr runs. (On HPCloud, the number of sec. groups is limited to about 7
> by default, so I easily run out of these (increasing the limit is taking
> time).)
>
> Based on http://code.google.com/p/**jclouds/issues/detail?id=814<http://code.google.com/p/jclouds/issues/detail?id=814>it looks like what I need is possible:
>
> client.**createSecurityGroupRuleAllowin**gSecurityGroupId(**securityGroup.getId(),
> ingress, securityGroup.getId());
>
>
> Paul
>
>
>
>
> On 20120817 16:46 , Paul Baclace wrote:
>
>> Thanks for the info.
>>
>> I would like to use the ec2 style extension on hpcloud, but I will see if
>> I get to that. I'm still making my custom image.
>>
>> Note: hpcloud supports ec2 style metadata like:
>> wget -q -O - http://169.254.169.254/latest/**meta-data/local-ipv4<http://169.254.169.254/latest/meta-data/local-ipv4>
>>
>>
>> Paul
>>
>> On 20120817 10:58 , Adrian Cole wrote:
>>
>>> Hi, Paul.
>>>
>>> Notes inline:
>>>
>>> On Thu, Aug 16, 2012 at 6:45 PM, Paul Baclace <pa...@gmail.com>
>>> wrote:
>>>
>>>> Adrian,
>>>>
>>>> Trying the patch did not consume too much time; I can use the
>>>> region/number
>>>> form for hw and images.
>>>>
>>> There's a new patch, btw. I found the issue you tripped over and
>>> fixed it in WHIRR-593-incTemplate.patch
>>>
>>>> Here is the next issue: start 2 zookeepers and they cannot communicate
>>>> because the jclouds-$cluster_name security group that Whirr created only
>>>> gave port 22 permission.
>>>>
>>>> a. how to specify ports to open between all the nodes created (rather
>>>> than
>>>> open to all of Internet or all of LAN)?
>>>>
>>> I don't think we have a setting for this, yet.
>>>
>>>> b. how to specify the name of an existing sec. group which also has the
>>>> "applies to all instances that are bound to it" property.
>>>>
>>> supporting openstack means adding support for their security group
>>> extension. I've made note of it here:
>>> https://issues.apache.org/**jira/browse/WHIRR-118<https://issues.apache.org/jira/browse/WHIRR-118>
>>>
>>> I suppose another way would be to use "openstack-nova-ec2" provider
>>> and specify hp as the endpoint. This could possibly work, and then
>>> you can use EC2 security group properties.
>>>
>>>> I hope this can be done without resorting to (n^2 - n)/2 distinct port
>>>> permissions.
>>>>
>>>>
>>>> Paul
>>>>
>>>>
>>>> On 20120816 7:25 , Adrian Cole wrote:
>>>>
>>>>> Hi, Paul.
>>>>>
>>>>> Sorry the patch didn't work for you, but glad to hear you discovered
>>>>> the params you need. I'll try *exactly* what you did and not ask you
>>>>> to test another patch until that works :)
>>>>>
>>>>> -A
>>>>>
>>>>> On Thu, Aug 16, 2012 at 2:49 AM, Paul Baclace <pa...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> P.s. if you go to the jclouds-cli repo on github, you can enumerate
>>>>>>> the
>>>>>>> IDs you need.
>>>>>>>
>>>>>> Good to know. I will try this.
>>>>>>
>>>>>> Using the latest WHIRR-593 patch for jclouds 1.5.0b10, I can start a
>>>>>> 64bit
>>>>>> "noop" machine which ends up as xsmall and ubuntu 10.04 despite these
>>>>>> config
>>>>>> props:
>>>>>>
>>>>>> # a well known image ubuntu 12.04 as listed in hpcloud console (image
>>>>>> id
>>>>>> is
>>>>>> "not found"):
>>>>>> whirr.image-id=8419
>>>>>>
>>>>>> # id not found:
>>>>>> whirr.hardware-id=standard.**small
>>>>>>
>>>>>> # id not found:
>>>>>> whirr.hardware-id=101
>>>>>>
>>>>>> # changing the minRam to 2048 did not force a small machine:
>>>>>>
>>>>>> whirr.template=osFamily=**UBUNTU,osVersionMatches=12.04,**os64Bit=true,minRam=2048
>>>>>>
>>>>>>
>>>>>> I *was* able to start a small machine using the region and the
>>>>>> openstack
>>>>>> machine code like this:
>>>>>> whirr.hardware-id=az-1.region-**a.geo-1/101
>>>>>>
>>>>>> Next I need to find the exact form for whirr.image-id so I can use a
>>>>>> custom
>>>>>> image (perhaps region is needed in the specifier or a full URL).
>>>>>>
>>>>>>
>>>>>> Paul
>>>>>>
>>>>>> On 20120815 18:29 , Paul Baclace wrote:
>>>>>>
>>>>>>> I built Whirr from trunk svn r1373566. The machine running whirr
>>>>>>> will be
>>>>>>> ubuntu 12.04 (I hope that will work.) The spawned cluster nodes can
>>>>>>> be
>>>>>>> ubuntu 10.04, but I thought CDH 4.0.1 specified that it worked with
>>>>>>> 12.04.
>>>>>>>
>>>>>>> Normally, I prepare a custom os image (based on a whirr cluster) to
>>>>>>> speed
>>>>>>> up provisioning. Will that work on HPCloud?
>>>>>>>
>>>>>>>
>>>>>>> Paul
>>>>>>>
>>>>>>> On 20120815 17:59 , Adrian Cole wrote:
>>>>>>>
>>>>>>>> The problem is that 10.04 is literally hardcoded. Before the above
>>>>>>>> patch,
>>>>>>>> there's no way to change that without knowing IDs.
>>>>>>>> On Aug 15, 2012 5:57 PM, "Andrew Bayer" <an...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> ...and you don't actually have to specify the OS stuff at all. Just
>>>>>>>>> whirr.hardware-min-ram=1024 or whatever, and it'll automatically
>>>>>>>>> pick
>>>>>>>>> an
>>>>>>>>> Ubuntu 10.04 image for you.
>>>>>>>>>
>>>>>>>>> A.
>>>>>>>>>
>>>>>>>>> On Wed, Aug 15, 2012 at 5:54 PM, Andrew Bayer <
>>>>>>>>> andrew.bayer@gmail.com
>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>> You'll actually have to do some manual tweaks due to HP Cloud
>>>>>>>>>> public
>>>>>>>>>> IPs
>>>>>>>>>> not having reverse-resolvable IP addresses - you'll need to
>>>>>>>>>> manually
>>>>>>>>>> go
>>>>>>>>>> onto each of your hosts and change any references to hostnames
>>>>>>>>>> they
>>>>>>>>>> put
>>>>>>>>>>
>>>>>>>>> in
>>>>>>>>>
>>>>>>>>>> there to the IP instead.
>>>>>>>>>>
>>>>>>>>>> A.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Aug 15, 2012 at 5:53 PM, Adrian Cole <ad...@jclouds.org>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi, Paul.
>>>>>>>>>>>
>>>>>>>>>>> Here's easiest way
>>>>>>>>>>>
>>>>>>>>>>> Apply latest patch here
>>>>>>>>>>> https://issues.apache.org/**jira/browse/WHIRR-593<https://issues.apache.org/jira/browse/WHIRR-593>
>>>>>>>>>>>
>>>>>>>>>>> Set
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> whirr.template=osFamily=**UBUNTU,osVersionMatches=12.04,**os64Bit=true,minRam=1024
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Should be good. If works, ping the issue and I can help commit it
>>>>>>>>>>> tonight.
>>>>>>>>>>>
>>>>>>>>>>> -A
>>>>>>>>>>> On Aug 15, 2012 5:45 PM, "Paul Baclace" <pa...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> I need to get Whirr 0.8.0 running on HPCloud *today* so I can
>>>>>>>>>>>> help
>>>>>>>>>>>>
>>>>>>>>>>> people
>>>>>>>>>>>
>>>>>>>>>>>> generate
>>>>>>>>>>>> clusters for the ACM local chapter (sfbayacm.org) hackathon on
>>>>>>>>>>>> Aug.
>>>>>>>>>>>>
>>>>>>>>>>> 18
>>>>>>>>>
>>>>>>>>>> sponsored in part by HPCloud.
>>>>>>>>>>>>
>>>>>>>>>>>> Because of WHIRR-593, I built Whirr from trunk.
>>>>>>>>>>>>
>>>>>>>>>>>> My plan is to use Ubuntu 12.04 LTS Precise, 64-bit.
>>>>>>>>>>>>
>>>>>>>>>>>> whirr.provider=hpcloud-compute
>>>>>>>>>>>>
>>>>>>>>>>>> How does one find the possible values for:
>>>>>>>>>>>>
>>>>>>>>>>>> whirr.image-id // I assume numbers seen in HP web console work
>>>>>>>>>>>> here
>>>>>>>>>>>> whirr.hardware-id // use number from openstack or names like
>>>>>>>>>>>> standard.medium ?
>>>>>>>>>>>> whirr.location-id // what is proper form here?
>>>>>>>>>>>>
>>>>>>>>>>>> and any other property settings I need to know about for
>>>>>>>>>>>> HPCloud?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>> Paul
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>
>
Re: Whirr on HPCloud
Posted by Steve Loughran <st...@hortonworks.com>.
On 15 March 2013 23:21, Paul Baclace <pa...@gmail.com> wrote:
> In Aug. 2012, HPCloud did not support the necessary security group
> features (symbolic instance membership), so I used manually-created CIDR
> /32 permissions. This could have been automated, but I did not have time to
> do that.
>
> When using OpenStack, it is important to be aware that any particular IaaS
> offering can have a different mix of version levels of the services. So one
> needs to know the implemented major version levels of nova, swift, quantum,
> keystone, glance, and cinder.
>
>
> Paul
be nice if there was a URL that would give you these details, and even
things like
swift: {
api: 0.2
implementation {
platform: swift:
version 0.2
}
}
which would distinguish it from, say
implementation {
platform: Glustre:
version 6.4
}
Re: Whirr on HPCloud
Posted by Paul Baclace <pa...@gmail.com>.
In Aug. 2012, HPCloud did not support the necessary security group
features (symbolic instance membership), so I used manually-created CIDR
/32 permissions. This could have been automated, but I did not have time
to do that.
When using OpenStack, it is important to be aware that any particular
IaaS offering can have a different mix of version levels of the
services. So one needs to know the implemented major version levels of
nova, swift, quantum, keystone, glance, and cinder.
Paul
On 20130315 13:20 , Andrew Bayer wrote:
> Not to engage in thread necromancy, but did you ever manage to get security
> groups on hp cloud working with whirr? =)
>
> A.
>
> On Sun, Aug 26, 2012 at 6:12 PM, Paul Baclace <pa...@gmail.com>wrote:
>
>>> From: Adrian Cole
>>> I suppose another way would be to use "openstack-nova-ec2" provider
>>> and specify hp as the endpoint. This could possibly work, and then
>>> you can use EC2 security group properties.
>> I am able to get an HP AUTH token with curl and use it to list sec.
>> groups, am not sure how to pass that to jclouds, so Whirr gets auth failure
>> just looking up images. I tried setting whirr.credential to the HP AUTH
>> token and also tried using the secret_key.
>>
>> Workaround would be to supply Whirr with a pre-exisiting security group to
>> use, then I can modify the sec. group with a w.x.y.z/32, CIDR list after
>> Whirr runs. (On HPCloud, the number of sec. groups is limited to about 7
>> by default, so I easily run out of these (increasing the limit is taking
>> time).)
>>
>> Based on http://code.google.com/p/**jclouds/issues/detail?id=814<http://code.google.com/p/jclouds/issues/detail?id=814>it looks like what I need is possible:
>>
>> client.**createSecurityGroupRuleAllowin**gSecurityGroupId(**securityGroup.getId(),
>> ingress, securityGroup.getId());
>>
>>
>> Paul
>>
>>
>>
>>
>> On 20120817 16:46 , Paul Baclace wrote:
>>
>>> Thanks for the info.
>>>
>>> I would like to use the ec2 style extension on hpcloud, but I will see if
>>> I get to that. I'm still making my custom image.
>>>
>>> Note: hpcloud supports ec2 style metadata like:
>>> wget -q -O - http://169.254.169.254/latest/**meta-data/local-ipv4<http://169.254.169.254/latest/meta-data/local-ipv4>
>>>
>>>
>>> Paul
>>>
>>> On 20120817 10:58 , Adrian Cole wrote:
>>>
>>>> Hi, Paul.
>>>>
>>>> Notes inline:
>>>>
>>>> On Thu, Aug 16, 2012 at 6:45 PM, Paul Baclace <pa...@gmail.com>
>>>> wrote:
>>>>
>>>>> Adrian,
>>>>>
>>>>> Trying the patch did not consume too much time; I can use the
>>>>> region/number
>>>>> form for hw and images.
>>>>>
>>>> There's a new patch, btw. I found the issue you tripped over and
>>>> fixed it in WHIRR-593-incTemplate.patch
>>>>
>>>>> Here is the next issue: start 2 zookeepers and they cannot communicate
>>>>> because the jclouds-$cluster_name security group that Whirr created only
>>>>> gave port 22 permission.
>>>>>
>>>>> a. how to specify ports to open between all the nodes created (rather
>>>>> than
>>>>> open to all of Internet or all of LAN)?
>>>>>
>>>> I don't think we have a setting for this, yet.
>>>>
>>>>> b. how to specify the name of an existing sec. group which also has the
>>>>> "applies to all instances that are bound to it" property.
>>>>>
>>>> supporting openstack means adding support for their security group
>>>> extension. I've made note of it here:
>>>> https://issues.apache.org/**jira/browse/WHIRR-118<https://issues.apache.org/jira/browse/WHIRR-118>
>>>>
>>>> I suppose another way would be to use "openstack-nova-ec2" provider
>>>> and specify hp as the endpoint. This could possibly work, and then
>>>> you can use EC2 security group properties.
>>>>
>>>>> I hope this can be done without resorting to (n^2 - n)/2 distinct port
>>>>> permissions.
>>>>>
>>>>>
>>>>> Paul
>>>>>
>>>>>
>>>>> On 20120816 7:25 , Adrian Cole wrote:
>>>>>
>>>>>> Hi, Paul.
>>>>>>
>>>>>> Sorry the patch didn't work for you, but glad to hear you discovered
>>>>>> the params you need. I'll try *exactly* what you did and not ask you
>>>>>> to test another patch until that works :)
>>>>>>
>>>>>> -A
>>>>>>
>>>>>> On Thu, Aug 16, 2012 at 2:49 AM, Paul Baclace <pa...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> P.s. if you go to the jclouds-cli repo on github, you can enumerate
>>>>>>>> the
>>>>>>>> IDs you need.
>>>>>>>>
>>>>>>> Good to know. I will try this.
>>>>>>>
>>>>>>> Using the latest WHIRR-593 patch for jclouds 1.5.0b10, I can start a
>>>>>>> 64bit
>>>>>>> "noop" machine which ends up as xsmall and ubuntu 10.04 despite these
>>>>>>> config
>>>>>>> props:
>>>>>>>
>>>>>>> # a well known image ubuntu 12.04 as listed in hpcloud console (image
>>>>>>> id
>>>>>>> is
>>>>>>> "not found"):
>>>>>>> whirr.image-id=8419
>>>>>>>
>>>>>>> # id not found:
>>>>>>> whirr.hardware-id=standard.**small
>>>>>>>
>>>>>>> # id not found:
>>>>>>> whirr.hardware-id=101
>>>>>>>
>>>>>>> # changing the minRam to 2048 did not force a small machine:
>>>>>>>
>>>>>>> whirr.template=osFamily=**UBUNTU,osVersionMatches=12.04,**os64Bit=true,minRam=2048
>>>>>>>
>>>>>>>
>>>>>>> I *was* able to start a small machine using the region and the
>>>>>>> openstack
>>>>>>> machine code like this:
>>>>>>> whirr.hardware-id=az-1.region-**a.geo-1/101
>>>>>>>
>>>>>>> Next I need to find the exact form for whirr.image-id so I can use a
>>>>>>> custom
>>>>>>> image (perhaps region is needed in the specifier or a full URL).
>>>>>>>
>>>>>>>
>>>>>>> Paul
>>>>>>>
>>>>>>> On 20120815 18:29 , Paul Baclace wrote:
>>>>>>>
>>>>>>>> I built Whirr from trunk svn r1373566. The machine running whirr
>>>>>>>> will be
>>>>>>>> ubuntu 12.04 (I hope that will work.) The spawned cluster nodes can
>>>>>>>> be
>>>>>>>> ubuntu 10.04, but I thought CDH 4.0.1 specified that it worked with
>>>>>>>> 12.04.
>>>>>>>>
>>>>>>>> Normally, I prepare a custom os image (based on a whirr cluster) to
>>>>>>>> speed
>>>>>>>> up provisioning. Will that work on HPCloud?
>>>>>>>>
>>>>>>>>
>>>>>>>> Paul
>>>>>>>>
>>>>>>>> On 20120815 17:59 , Adrian Cole wrote:
>>>>>>>>
>>>>>>>>> The problem is that 10.04 is literally hardcoded. Before the above
>>>>>>>>> patch,
>>>>>>>>> there's no way to change that without knowing IDs.
>>>>>>>>> On Aug 15, 2012 5:57 PM, "Andrew Bayer" <an...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> ...and you don't actually have to specify the OS stuff at all. Just
>>>>>>>>>> whirr.hardware-min-ram=1024 or whatever, and it'll automatically
>>>>>>>>>> pick
>>>>>>>>>> an
>>>>>>>>>> Ubuntu 10.04 image for you.
>>>>>>>>>>
>>>>>>>>>> A.
>>>>>>>>>>
>>>>>>>>>> On Wed, Aug 15, 2012 at 5:54 PM, Andrew Bayer <
>>>>>>>>>> andrew.bayer@gmail.com
>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>> You'll actually have to do some manual tweaks due to HP Cloud
>>>>>>>>>>> public
>>>>>>>>>>> IPs
>>>>>>>>>>> not having reverse-resolvable IP addresses - you'll need to
>>>>>>>>>>> manually
>>>>>>>>>>> go
>>>>>>>>>>> onto each of your hosts and change any references to hostnames
>>>>>>>>>>> they
>>>>>>>>>>> put
>>>>>>>>>>>
>>>>>>>>>> in
>>>>>>>>>>
>>>>>>>>>>> there to the IP instead.
>>>>>>>>>>>
>>>>>>>>>>> A.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Aug 15, 2012 at 5:53 PM, Adrian Cole <ad...@jclouds.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi, Paul.
>>>>>>>>>>>> Here's easiest way
>>>>>>>>>>>>
>>>>>>>>>>>> Apply latest patch here
>>>>>>>>>>>> https://issues.apache.org/**jira/browse/WHIRR-593<https://issues.apache.org/jira/browse/WHIRR-593>
>>>>>>>>>>>>
>>>>>>>>>>>> Set
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> whirr.template=osFamily=**UBUNTU,osVersionMatches=12.04,**os64Bit=true,minRam=1024
>>>>>>>>>>
>>>>>>>>>>> Should be good. If works, ping the issue and I can help commit it
>>>>>>>>>>>> tonight.
>>>>>>>>>>>>
>>>>>>>>>>>> -A
>>>>>>>>>>>> On Aug 15, 2012 5:45 PM, "Paul Baclace" <pa...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I need to get Whirr 0.8.0 running on HPCloud *today* so I can
>>>>>>>>>>>>> help
>>>>>>>>>>>>>
>>>>>>>>>>>> people
>>>>>>>>>>>>
>>>>>>>>>>>>> generate
>>>>>>>>>>>>> clusters for the ACM local chapter (sfbayacm.org) hackathon on
>>>>>>>>>>>>> Aug.
>>>>>>>>>>>>>
>>>>>>>>>>>> 18
>>>>>>>>>>> sponsored in part by HPCloud.
>>>>>>>>>>>>> Because of WHIRR-593, I built Whirr from trunk.
>>>>>>>>>>>>>
>>>>>>>>>>>>> My plan is to use Ubuntu 12.04 LTS Precise, 64-bit.
>>>>>>>>>>>>>
>>>>>>>>>>>>> whirr.provider=hpcloud-compute
>>>>>>>>>>>>>
>>>>>>>>>>>>> How does one find the possible values for:
>>>>>>>>>>>>>
>>>>>>>>>>>>> whirr.image-id // I assume numbers seen in HP web console work
>>>>>>>>>>>>> here
>>>>>>>>>>>>> whirr.hardware-id // use number from openstack or names like
>>>>>>>>>>>>> standard.medium ?
>>>>>>>>>>>>> whirr.location-id // what is proper form here?
>>>>>>>>>>>>>
>>>>>>>>>>>>> and any other property settings I need to know about for
>>>>>>>>>>>>> HPCloud?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Paul
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>