You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@deltacloud.apache.org by lu...@redhat.com on 2012/08/21 01:28:07 UTC
EC2: launch instances into a VPC
As recently discussed, these patches make it possible to launch instances
into a subnet in a VPC in EC2.
Subnets appear as new realms with the name AZ:SN where AZ is the
availability zone (e.g., us-east-1a) to which the subnet is attached, and
SN is the subnet ID.
When such a realm is supplied to the create instances call, the SubnetID is
sent to EC2, placing the instane in that subnet.
Note that you can not provide a security group when launching into a VPC,
which makes the current HTML UI a little awkward, in that you have to
uncheck the 'default' security group in the UI under 'Additional
Parameters'
The VPC functionality requires some additional mappings from the AWS gem,
which I have sent upstream[1] - until they are in a released AWS gem, I am
including a monkey patch that puts them into the AWS::Ec2 class.
David
[1] https://github.com/appoxy/aws/pull/116
[PATCH 1/2] EC2: list subnets as realms
Posted by lu...@redhat.com.
From: David Lutterkort <lu...@redhat.com>
Subnets (from VPC's) are listed as realms.
Note that this patch monkey patches the aws gem; the corresponding patch
has been sent upstream, it is included here to make testing the patch easier.
---
.../deltacloud/drivers/ec2/aws_vpc_monkey_patch.rb | 294 ++++++++++++++++++++
server/lib/deltacloud/drivers/ec2/ec2_driver.rb | 29 ++-
2 files changed, 319 insertions(+), 4 deletions(-)
create mode 100644 server/lib/deltacloud/drivers/ec2/aws_vpc_monkey_patch.rb
diff --git a/server/lib/deltacloud/drivers/ec2/aws_vpc_monkey_patch.rb b/server/lib/deltacloud/drivers/ec2/aws_vpc_monkey_patch.rb
new file mode 100644
index 0000000..3f1d491
--- /dev/null
+++ b/server/lib/deltacloud/drivers/ec2/aws_vpc_monkey_patch.rb
@@ -0,0 +1,294 @@
+# This is a copy of code that has been submitted upstream
+# https://github.com/appoxy/aws/pull/116
+#
+# If you make changes here, make sure they go upstream, too
+
+unless Aws::Ec2::method_defined?(:create_vpc)
+ class Aws::Ec2
+ #-----------------------------------------------------------------
+ # VPC related
+ #-----------------------------------------------------------------
+
+ # Create VPC
+ # http://docs.amazonwebservices.com/AWSEC2/latest/APIReference/ApiReference-query-CreateVpc.html
+ #
+ # ec2.create_vpc("10.0.0.0/16")
+ # FIXME: EVen though the EC2 docs describe the parameter instanceTenancy,
+ # I could not get it to recognize that
+ def create_vpc(cidr_block = "10.0.0.0/16")
+ params = { "CidrBlock" => cidr_block }
+ link = generate_request("CreateVpc", params)
+ request_info(link, QEc2VpcsParser.new("vpc", :logger => @logger))
+ rescue Exception
+ on_exception
+ end
+
+
+ # Describe VPC's
+ # http://docs.amazonwebservices.com/AWSEC2/latest/APIReference/ApiReference-query-DescribeVpcs.html
+ #
+ # ec2.describe_vpcs
+ # ec2.describe_vpcs(vpcId1, vpcId2, 'Filter.1.Name' => 'state', 'Filter.1.Value' = > 'pending', ...)
+ def describe_vpcs(*args)
+ if args.last.is_a?(Hash)
+ params = args.pop.dup
+ else
+ params = {}
+ end
+ 1.upto(args.size) { |i| params["VpcId.#{i}"] = args[i-1] }
+ link = generate_request("DescribeVpcs", params)
+ request_info(link, QEc2VpcsParser.new("item", :logger => @logger))
+ rescue Exception
+ on_exception
+ end
+
+ # Delete VPC
+ # http://docs.amazonwebservices.com/AWSEC2/latest/APIReference/ApiReference-query-DeleteVpc.html
+ #
+ # ec2.delete_vpc(vpc_id)
+ def delete_vpc(vpc_id)
+ params = { "VpcId" => vpc_id }
+ link = generate_request("DeleteVpc", params)
+ request_info(link, RightBoolResponseParser.new(:logger => @logger))
+ rescue Exception
+ on_exception
+ end
+
+ # Create subnet in a VPC
+ # http://docs.amazonwebservices.com/AWSEC2/latest/APIReference/ApiReference-query-CreateSubnet.html
+ #
+ # ec2.create_subnet(vpc_id, cidr_block)
+ # ec2.create_subnet(vpc_id, cidr_block, availability_zone))
+ def create_subnet(vpc_id, cidr_block, availability_zone = nil)
+ params = { "VpcId" => vpc_id, "CidrBlock" => cidr_block }
+ params["AvailabilityZone"] = availability_zone if availability_zone
+ link = generate_request("CreateSubnet", params)
+ request_info(link, QEc2SubnetsParser.new("subnet", :logger => @logger))
+ rescue Exception
+ on_exception
+ end
+
+ # Describe subnets
+ # http://docs.amazonwebservices.com/AWSEC2/latest/APIReference/ApiReference-query-DescribeSubnets.html
+ #
+ # ec2.describe_subnets
+ # ecs.describe_subnets(subnetId1, SubnetId2, ...,
+ # 'Filter.1.Name' => 'state',
+ # 'Filter.1.Value.1' => 'pending',
+ # 'Filter.2.Name' => ...)
+ def describe_subnets(*args)
+ if args.last.is_a?(Hash)
+ params = args.pop.dup
+ else
+ params = {}
+ end
+ 1.upto(args.size) { |i| params["SubnetId.#{i}"] = args[i-1] }
+ link = generate_request("DescribeSubnets", params)
+ request_info(link, QEc2SubnetsParser.new("item", :logger => @logger))
+ rescue Exception
+ on_exception
+ end
+
+ # Delete Subnet
+ # http://docs.amazonwebservices.com/AWSEC2/latest/APIReference/ApiReference-query-DeleteSubnet.html
+ #
+ # ec2.delete_subnet(subnet_id)
+ def delete_subnet(subnet_id)
+ params = { "SubnetId" => subnet_id }
+ link = generate_request("DeleteSubnet", params)
+ request_info(link, RightBoolResponseParser.new(:logger => @logger))
+ rescue Exception
+ on_exception
+ end
+
+ # The only change in this class compared to upstream is
+ # that we parse out subnetId and vpcId
+ class QEc2DescribeInstancesParser < Aws::AwsParser #:nodoc:
+ def tagstart(name, attributes)
+ # DescribeInstances property
+ if (name == 'item' && @xmlpath == 'DescribeInstancesResponse/reservationSet') ||
+ # RunInstances property
+ (name == 'RunInstancesResponse')
+ @reservation = {:aws_groups => [],
+ :instances_set => []}
+
+ elsif (name == 'item') &&
+ # DescribeInstances property
+ (@xmlpath=='DescribeInstancesResponse/reservationSet/item/instancesSet' ||
+ # RunInstances property
+ @xmlpath=='RunInstancesResponse/instancesSet')
+ # the optional params (sometimes are missing and we dont want them to be nil)
+ @instance = {:aws_reason => '',
+ :dns_name => '',
+ :private_dns_name => '',
+ :ami_launch_index => '',
+ :ssh_key_name => '',
+ :aws_state => '',
+ :root_device_type => '',
+ :root_device_name => '',
+ :architecture => '',
+ :subnet_id => '',
+ :vpc_id => '',
+ :block_device_mappings => [],
+ :aws_product_codes => [],
+ :tags => {}}
+ end
+ end
+
+ def tagend(name)
+ case name
+ when 'rootDeviceType' then
+ @instance[:root_device_type] = @text
+ when 'architecture' then
+ @instance[:architecture] = @text
+ when 'rootDeviceName' then
+ @instance[:root_device_name] = @text
+ # reservation
+ when 'reservationId' then
+ @reservation[:aws_reservation_id] = @text
+ when 'ownerId' then
+ @reservation[:aws_owner] = @text
+ when 'groupId' then
+ @reservation[:aws_groups] << @text
+ # instance
+ when 'instanceId' then
+ @instance[:aws_instance_id] = @text
+ when 'imageId' then
+ @instance[:aws_image_id] = @text
+ when 'dnsName' then
+ @instance[:dns_name] = @text
+ when 'privateDnsName' then
+ @instance[:private_dns_name] = @text
+ when 'reason' then
+ @instance[:aws_reason] = @text
+ when 'keyName' then
+ @instance[:ssh_key_name] = @text
+ when 'amiLaunchIndex' then
+ @instance[:ami_launch_index] = @text
+ when 'code' then
+ @instance[:aws_state_code] = @text
+ when 'name' then
+ @instance[:aws_state] = @text
+ when 'productCode' then
+ @instance[:aws_product_codes] << @text
+ when 'instanceType' then
+ @instance[:aws_instance_type] = @text
+ when 'launchTime' then
+ @instance[:aws_launch_time] = @text
+ when 'kernelId' then
+ @instance[:aws_kernel_id] = @text
+ when 'ramdiskId' then
+ @instance[:aws_ramdisk_id] = @text
+ when 'platform' then
+ @instance[:aws_platform] = @text
+ when 'availabilityZone' then
+ @instance[:aws_availability_zone] = @text
+ when 'privateIpAddress' then
+ @instance[:aws_private_ip_address] = @text
+ when 'subnetId' then
+ @instance[:subnet_id] = @text
+ when 'vpcId' then
+ @instance[:vpc_id] = @text
+ when 'key' then
+ @tag_key = @text
+ when 'value' then
+ @tag_value = @text
+ when 'deviceName' then
+ @device_name = @text
+ when 'volumeId' then
+ @volume_id = @text
+ when 'state'
+ if @xmlpath == 'DescribeInstancesResponse/reservationSet/item/instancesSet/item/monitoring' || # DescribeInstances property
+ @xmlpath == 'RunInstancesResponse/instancesSet/item/monitoring' # RunInstances property
+ @instance[:monitoring_state] = @text
+ end
+ when 'item'
+ if @xmlpath=='DescribeInstancesResponse/reservationSet/item/instancesSet/item/tagSet' # Tags
+ @instance[:tags][@tag_key] = @tag_value
+ elsif @xmlpath == 'DescribeInstancesResponse/reservationSet/item/instancesSet/item/blockDeviceMapping' # Block device mappings
+ @instance[:block_device_mappings] << { @device_name => @volume_id }
+ elsif @xmlpath == 'DescribeInstancesResponse/reservationSet/item/instancesSet' || # DescribeInstances property
+ @xmlpath == 'RunInstancesResponse/instancesSet' # RunInstances property
+ @reservation[:instances_set] << @instance
+ elsif @xmlpath=='DescribeInstancesResponse/reservationSet' # DescribeInstances property
+ @result << @reservation
+ end
+ when 'RunInstancesResponse' then
+ @result << @reservation # RunInstances property
+ end
+ end
+
+ def reset
+ @result = []
+ end
+ end
+
+ #-----------------------------------------------------------------
+ # PARSERS: Vpc
+ #-----------------------------------------------------------------
+
+ class QEc2VpcsParser < Aws::AwsParser #:nodoc:
+ def initialize(wrapper, opts = {})
+ super(opts)
+ @wrapper = wrapper
+ end
+
+ def tagstart(name, attribute)
+ @vpc = {} if name == @wrapper
+ end
+
+ def tagend(name)
+ case name
+ when 'vpcId' then
+ @vpc[:vpc_id] = @text
+ when 'state' then
+ @vpc[:state] = @text
+ when 'cidrBlock' then
+ @vpc[:cidr_block] = @text
+ when 'dhcpOptionsId' then
+ @vpc[:dhcp_options_id] = @text
+ when @wrapper
+ @result << @vpc
+ end
+ end
+
+ def reset
+ @result = []
+ end
+ end
+
+ class QEc2SubnetsParser < Aws::AwsParser #:nodoc
+ def initialize(wrapper, opts = {})
+ super(opts)
+ @wrapper = wrapper
+ end
+
+ def tagstart(name, attribute)
+ @subnet = {} if name == @wrapper
+ end
+
+ def tagend(name)
+ case name
+ when 'subnetId' then
+ @subnet[:subnet_id] = @text
+ when 'state' then
+ @subnet[:state] = @text
+ when 'vpcId' then
+ @subnet[:vpc_id] = @text
+ when 'cidrBlock' then
+ @subnet[:cidr_block] = @text
+ when 'availableIpAddressCount' then
+ @subnet[:available_ip_address_count] = @text
+ when 'availabilityZone' then
+ @subnet[:availability_zone] = @text
+ when @wrapper
+ @result << @subnet
+ end
+ end
+
+ def reset
+ @result = []
+ end
+ end
+ end
+end
diff --git a/server/lib/deltacloud/drivers/ec2/ec2_driver.rb b/server/lib/deltacloud/drivers/ec2/ec2_driver.rb
index 8847034..31fcfa5 100644
--- a/server/lib/deltacloud/drivers/ec2/ec2_driver.rb
+++ b/server/lib/deltacloud/drivers/ec2/ec2_driver.rb
@@ -15,6 +15,8 @@
#
require 'aws'
+# Delete this once VPC support is merged upstream
+require_relative 'aws_vpc_monkey_patch'
require_relative '../../runner'
@@ -164,22 +166,35 @@ module Deltacloud
end
def realms(credentials, opts={})
+ # We have two different kinds of realms:
+ # (1) Availability Zones
+ # (2) Subnets in VPC's (scoped to an AZ)
+ # For the latter, the ID is AZ:SUBNET, and we can tell that we
+ # are looking at such a realm by checking if the id contains a colon
ec2 = new_client(credentials)
realms = []
safely do
if opts[:id] and !opts[:id].empty?
+ az, sn = opts[:id].split(":")
begin
- ec2.describe_availability_zones([opts[:id]]).collect do |realm|
- realms << convert_realm(realm) unless realm.empty?
+ if sn
+ subnet = ec2.describe_subnets(sn).first
+ realms << convert_realm(subnet) if subnet
+ else
+ ec2.describe_availability_zones(az).collect do |realm|
+ realms << convert_realm(realm) unless realm.empty?
+ end
end
rescue => e
raise e unless e.message =~ /Invalid availability zone/
realms = []
end
else
- ec2.describe_availability_zones.collect do |realm|
- realms << convert_realm(realm) unless realm.empty?
+ realms = ec2.describe_availability_zones.collect do |realm|
+ convert_realm(realm) unless realm.empty?
end
+ realms = realms +
+ ec2.describe_subnets.map { |sn| convert_realm(sn) }
end
end
realms
@@ -871,6 +886,12 @@ module Deltacloud
end
def convert_realm(realm)
+ # We also allow subnets as realms
+ if realm[:subnet_id]
+ realm[:zone_name] =
+ "#{realm[:availability_zone]}:#{realm[:subnet_id]}"
+ realm[:zone_state] = realm[:state]
+ end
Realm.new(
:id => realm[:zone_name],
:name => realm[:zone_name],
--
1.7.7.6
Re: EC2: launch instances into a VPC
Posted by Michal Fojtik <mf...@redhat.com>.
On Aug 21, 2012, at 1:28 AM, lutter@redhat.com wrote:
NACK :(
Unfortunately this patch set will break EC2 tests (rake test:drivers:ec2):
1) Error:
test_0002_must return list of realms(Ec2Driver Realms):
Deltacloud::ExceptionHandler::ProviderError: RequestExpired: Request has expired. Timestamp date is 2012-07-30T11:05:00.000Z
REQUEST=ec2.us-east-1.amazonaws.com:443/?AWSAccessKeyId=AKIAJYOQYLLOIWN5LQ3A&Action=DescribeSubnets&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2012-07-30T11%3A05%3A00.000Z&Version=2010-08-31&Signature=IiUb4TMnXSY2vdLZcsQ5PEHbIlpgfT1BUe7BeYcaf%2Fg%3D
REQUEST ID=d56ecde0-4ae0-4489-8cb3-b761bf42eb14
/Users/mfojtik/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/aws-2.5.6/lib/awsbase/awsbase.rb:572:in `request_info_impl'
/Users/mfojtik/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/aws-2.5.6/lib/ec2/ec2.rb:177:in `request_info'
/Users/mfojtik/code/core/server/lib/deltacloud/drivers/ec2/aws_vpc_monkey_patch.rb:87:in `describe_subnets'
/Users/mfojtik/code/core/server/lib/deltacloud/drivers/ec2/ec2_driver.rb:197:in `block in realms'
* This one looks like not recorded request, since it's touching EC2. The test should be 're-recorded' so it will
include this request as well.
2) Error:
test_0004_must allow to retrieve single realm(Ec2Driver Realms):
Deltacloud::ExceptionHandler::ProviderError: undefined method `to_a' for "us-east-1a":String
/Users/mfojtik/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/aws-2.5.6/lib/ec2/ec2.rb:1119:in `describe_availability_zones'
/Users/mfojtik/code/core/server/lib/deltacloud/drivers/ec2/ec2_driver.rb:184:in `block in realms'
/Users/mfojtik/code/core/server/lib/deltacloud/drivers/exceptions.rb:181:in `call'
/Users/mfojtik/code/core/server/lib/deltacloud/drivers/exceptions.rb:181:in `safely'
/Users/mfojtik/code/core/server/lib/deltacloud/drivers/ec2/ec2_driver.rb:176:in `realms'
/Users/mfojtik/code/core/server/lib/deltacloud/drivers/base_driver.rb:213:in `realm'
/Users/mfojtik/code/core/server/lib/deltacloud/api.rb:119:in `method_missing'
/Users/mfojtik/code/core/server/tests/drivers/ec2/realms_test.rb:37:in `block (2 levels) in <top (required)>'
3) Error:
test_0003_must allow to filter realms(Ec2Driver Realms):
Deltacloud::ExceptionHandler::ProviderError: undefined method `to_a' for "us-east-1a":String
/Users/mfojtik/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/aws-2.5.6/lib/ec2/ec2.rb:1119:in `describe_availability_zones'
/Users/mfojtik/code/core/server/lib/deltacloud/drivers/ec2/ec2_driver.rb:184:in `block in realms'
/Users/mfojtik/code/core/server/lib/deltacloud/drivers/exceptions.rb:181:in `call'
/Users/mfojtik/code/core/server/lib/deltacloud/drivers/exceptions.rb:181:in `safely'
/Users/mfojtik/code/core/server/lib/deltacloud/drivers/ec2/ec2_driver.rb:176:in `realms'
/Users/mfojtik/code/core/server/lib/deltacloud/api.rb:119:in `method_missing'
/Users/mfojtik/code/core/server/tests/drivers/ec2/realms_test.rb:29:in `block (2 levels) in <top (required)>
* No idea about this two :-)
But the code looks great, no inline comments from me.
Also you should include some tests for EC2 driver for this feature as well.
-- Michal
> As recently discussed, these patches make it possible to launch instances
> into a subnet in a VPC in EC2.
>
> Subnets appear as new realms with the name AZ:SN where AZ is the
> availability zone (e.g., us-east-1a) to which the subnet is attached, and
> SN is the subnet ID.
>
> When such a realm is supplied to the create instances call, the SubnetID is
> sent to EC2, placing the instane in that subnet.
>
> Note that you can not provide a security group when launching into a VPC,
> which makes the current HTML UI a little awkward, in that you have to
> uncheck the 'default' security group in the UI under 'Additional
> Parameters'
>
> The VPC functionality requires some additional mappings from the AWS gem,
> which I have sent upstream[1] - until they are in a released AWS gem, I am
> including a monkey patch that puts them into the AWS::Ec2 class.
>
> David
>
> [1] https://github.com/appoxy/aws/pull/116
Michal Fojtik
http://deltacloud.org
mfojtik@redhat.com
[PATCH 2/2] EC2: allow launching instances into a subnet
Posted by lu...@redhat.com.
From: David Lutterkort <lu...@redhat.com>
---
server/lib/deltacloud/drivers/ec2/ec2_driver.rb | 15 +++++++++++++--
1 files changed, 13 insertions(+), 2 deletions(-)
diff --git a/server/lib/deltacloud/drivers/ec2/ec2_driver.rb b/server/lib/deltacloud/drivers/ec2/ec2_driver.rb
index 31fcfa5..b275f98 100644
--- a/server/lib/deltacloud/drivers/ec2/ec2_driver.rb
+++ b/server/lib/deltacloud/drivers/ec2/ec2_driver.rb
@@ -263,8 +263,15 @@ module Deltacloud
if opts[:metrics] and !opts[:metrics].empty?
instance_options[:monitoring_enabled] = true
end
+ if opts[:realm_id]
+ az, sn = opts[:realm_id].split(":")
+ if sn
+ instance_options[:subnet_id] = sn
+ else
+ instance_options[:availability_zone] = az
+ end
+ end
instance_options[:key_name] = opts[:keyname] if opts[:keyname]
- instance_options[:availability_zone] = opts[:realm_id] if opts[:realm_id]
instance_options[:instance_type] = opts[:hwp_id] if opts[:hwp_id] && opts[:hwp_id].length > 0
firewalls = opts.inject([]){|res, (k,v)| res << v if k =~ /firewalls\d+$/; res}
instance_options[:group_ids] = firewalls unless firewalls.empty?
@@ -919,6 +926,10 @@ module Deltacloud
if instance[:aws_instance_type] == "t1.micro"
inst_profile_opts[:hwp_architecture]=instance[:architecture]
end
+ realm_id = instance[:aws_availability_zone]
+ unless instance[:subnet_id].empty?
+ realm_id = "#{realm_id}:#{instance[:subnet_id]}"
+ end
Instance.new(
:id => instance[:aws_instance_id],
:name => instance[:aws_image_id],
@@ -929,7 +940,7 @@ module Deltacloud
:keyname => instance[:ssh_key_name],
:launch_time => instance[:aws_launch_time],
:instance_profile => InstanceProfile.new(instance[:aws_instance_type], inst_profile_opts),
- :realm_id => instance[:aws_availability_zone],
+ :realm_id => realm_id,
:public_addresses => [InstanceAddress.new(instance[:dns_name], :type => :hostname)],
:private_addresses => [InstanceAddress.new(instance[:private_dns_name], :type => :hostname)],
:firewalls => instance[:aws_groups],
--
1.7.7.6