Security Groups in Hadoop-ec2

[Tim Hawkins <ti...@bejant.com>]

I have been playing with hadoop-ec2 (src/contrib/ec2) and have found a  
minor problem

When a cluster is launched, it creates two security groups and cross  
links them, ie for cluster-name xxxx it creates EC2 security groups  
xxxx and xxxx-master , the cross-linking is i believe to allow all the  
masters and slaves to be able to communicate with each other.

Unfortunately this means that the security groups are now mutually  
dependant on each other, and the amazon API will no longer allow them  
to be deleted, none of the GUI tools (rightscale, elasticFox, and  
consol.aws) or the command-line tools seem to be able to remove the  
security groups either, presumably because they too are dependant on  
the API.

I believe the solution would be to create 3 security groups, not two,  
xxxx, xxxx-slave and xxxx-master , and only inherit permissions from  
xxxx into the other two, which would achieve the same result but be  
more "friendly". It would also potentially offer the ability for  
multiple clusters to share base security descriptors with other  
subsystems without having to open publicly accessible holes, by  
allowing the name of xxxx to be set independent of the cluster-name,  
allowing it to be shared as a base.

I can make this change to the scripts and test it on our set-up, but  
am not sure how to contribute the changes back to ensure that this  
problem does not effect others.

Re: Security Groups in Hadoop-ec2

[Tom White <to...@gmail.com>]

Hi Tim,

You can use ec2-revoke to disassociate the groups from each other,
then delete the groups. In fact, the "hadoop-ec2 delete-cluster"
command does exactly this. Does this solve your problem?

Inheriting from a base group might be a useful enhancement - would you
like to start a Jira for this?

Thanks,

Tom

BTW questions about Hadoop on EC2 are best posted to core-user.

On Fri, Jan 16, 2009 at 3:56 PM, Tim Hawkins <ti...@bejant.com> wrote:
> I have been playing with hadoop-ec2 (src/contrib/ec2) and have found a minor
> problem
>
> When a cluster is launched, it creates two security groups and cross links
> them, ie for cluster-name xxxx it creates EC2 security groups xxxx and
> xxxx-master , the cross-linking is i believe to allow all the masters and
> slaves to be able to communicate with each other.
>
> Unfortunately this means that the security groups are now mutually dependant
> on each other, and the amazon API will no longer allow them to be deleted,
> none of the GUI tools (rightscale, elasticFox, and consol.aws) or the
> command-line tools seem to be able to remove the security groups either,
> presumably because they too are dependant on the API.
>
> I believe the solution would be to create 3 security groups, not two, xxxx,
> xxxx-slave and xxxx-master , and only inherit permissions from xxxx into the
> other two, which would achieve the same result but be more "friendly". It
> would also potentially offer the ability for multiple clusters to share base
> security descriptors with other subsystems without having to open publicly
> accessible holes, by allowing the name of xxxx to be set independent of the
> cluster-name, allowing it to be shared as a base.
>
> I can make this change to the scripts and test it on our set-up, but am not
> sure how to contribute the changes back to ensure that this problem does not
> effect others.
>
>
>
>
>