You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@vcl.apache.org by bm...@apache.org on 2009/02/01 22:12:29 UTC
svn commit: r739839 - in
/incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning: esx.README
esx.pm
Author: bmbouter
Date: Sun Feb 1 21:12:28 2009
New Revision: 739839
URL: http://svn.apache.org/viewvc?rev=739839&view=rev
Log:
VCL-29
This patch reworks how the module discovers the private IP address of VMs that it provisions to ESX and ESX 3i hosts. Specifically, it implements the following new work flow:
1) Create the VM on the hypervisor, but DO NOT specify any mac addresses
2) Power on the VM
3) Ask the hypervisor, what the MAC address is of the eth0 (private) interface of the vm just provisioned
4) Watch the arp table of the management module for the dhcp address handed back to the VM when it's eth0 does the required dhcp request.
5) Delete the old entry from /etc/hosts for the VM
6) Add the new entry to /etc/hosts so that VCL can resolve this VM's hostname for account setup, etc...
This simplifies the setup significantly by removing the following three manual, per-vm steps:
1) Manual setting of eth0 mac address in database
2) Manual configuration of that same eth0 mac in the the management module's dhcpd.conf
3) Manual addition of IP/hostname pair to /etc/hosts file
The documentation (esx.README) has been simplified and updated as a result
Modified:
incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.README
incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.pm
Modified: incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.README
URL: http://svn.apache.org/viewvc/incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.README?rev=739839&r1=739838&r2=739839&view=diff
==============================================================================
--- incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.README (original)
+++ incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.README Sun Feb 1 21:12:28 2009
@@ -1,19 +1,18 @@
Pre-Installation:
-You need to install the VMware VI Perl toolkit installed on the VCL server
+You need to install the VMware VI Perl toolkit installed on any management node using the esx module
-Each vmware esx hypervisor requires a mounted NFS datastore named 'VCL'
+Each vmware ESX hypervisor requires a mounted NFS datastore named 'VCL'
-This same datastore needs to be mounted on the VCL server at /mnt/vcl/
+Mount this same datastore on the VCL management node /mnt/vcl/
This NFS export needs to be exported (from the NFS server) at /mnt/export/
-This same datastore requires the following structure.
-
+Populate this datastore with the following structure
[VCL]
/inuse (required)
/golden (required)
- /imagename
+ /imagename (a folder in golden for each image in the library)
imagename.vmdk
imagename-flat.vmdk
...
@@ -23,25 +22,41 @@
ssh must be set to start on boot
ssh keys must be setup so that VCL can ssh into it
The image needs two NICs (eth0 is private, eth1 is public)
- The IP addressing of the private (eth1) interface must be set to DHCP on boot
+ The IP addressing of the private (eth0) interface must be set to DHCP on boot
+
+The VCL management node acts as a dhcp server for the private (eth0) NICs of the virtual machines that will be provisioned. Configure your management node's private interface eth1 (10.0.0.1) to hand out valid IP addresses on the network 10.0.0.0/16 by configuring your /etc/dhcpd.conf and turning on dhcp. Use the following entry as a guide:
+ subnet 10.0.0.0 netmask 255.255.0.0 {
+ max-lease-time 43200;
+ min-lease-time 43200;
+ default-lease-time 43200;
+ option routers 10.0.0.1;
+ option subnet-mask 255.255.0.0;
+ option nis-domain "NA";
+ option domain-name "vcl.internal";
+ option domain-name-servers 10.0.0.1;
+ option nis-servers noip;
+ option time-offset -5;
+ range 10.0.0.2 10.0.255.255;
-VCL expects the blades to have a consistant private IP address. Go configure dhcpd.conf and in /etc/hosts manually
-Customize the module:
+Set a few variables and install the module:
This module expects the following three variables to be hard-coded at the top of the esx.pm file. Remember to character escape any special characters.
$vmhost_username, $vmhost_password, $datastore_ip
-A note about datastore_ip. Because the module ssh's to the storage system directly instead of doing a cp over nfs, you need to set the IP of the datastore here. The datastore machine should be setup with SSH keys from the VCL ssh keys set as IDENTITY_blade_linux in the VCL conf file.
+A note about datastore_ip. Because the module (from the management node) ssh's to the storage system directly instead of doing a cp over nfs, you need to set the IP of the datastore here. The machine exporting the NFS datastore should be setup with SSH keys from the VCL ssh keys. This is set in the VCL conf file as the IDENTITY_blade_linux variable.
+
+Place the module (esx.pm) in your management node's lib/VCL/Module/Provisioning/ directory
+
-Install the module:
Do these once:
+
1) Insert a row into the module table with `perlpackage` equal to 'VCL::Module::Provisioning::esx'
-2) Insert a row into the provisioning table with `moduleid` equal to the id of the entry from step 1 in the module table
+2) Insert a row into the provisioning table with `moduleid` equal to the id of the entry from step 1 (inserting into the module table)
3) Insert a record into the 'vmprofile' table with the following guidelines
profilename: VMware ESX SAN
vmtypeid: link to anything valid ID from the vmtype table.
@@ -49,38 +64,39 @@
virtualswitch0: <name of your private virtual machine port group>
virtualswitch1: <name of your public virtual machine port group>
-Make VCL aware of your ESX hosts:
-1) For each esx hypervisor, which VCL should provisong virtual machine to, create an entry for that hypervisor by going to Manage Computers -> Edit Computer Information -> Add
+
+
+For each ESX hypervisor which VCL will provision machines on, do the following:
+
+1) Create an entry for that hypervisor by going to Manage Computers -> Edit Computer Information -> Add
Hostname <your ESX's hostname>
IP <your ESX's IP>
Type "blade"
Provisioning Engine: This doesn't matter
Computer Groups: Don't add the hypervisors to any computer groups
-2) For each esx hypervisor manually create an entry in the 'vmhost' database table with the following guidelines:
+2) For each ESX hypervisor, manually create an entry in the 'vmhost' database table with the following guidelines:
computerid: the id of the computer created in step 1
- vmlimit: the max number of vms to have on this hypervisor
- vmprofileid: the id of the vmprofile entry created in the "Do these once" secion step 3
+ vmlimit: the max number of vms to have on this hypervisor (ie: 5)
+ vmprofileid: the id of the vmprofile entry created in step 3 of the above section labeled "Do these once"
+
+
+
+Assigning VMs to your hypervisors:
-Create a virtual machine placeholder for each VM you would like to concurrently run. Do so by going to Manage Computers -> Edit Computer Information -> Add
+1) Create a virtual machine placeholder for each VM you would like to concurrently run. Do so by going to Manage Computers -> Edit Computer Information -> Add
Hostname: <the hostname of the vm when it is provisioned and turned on>
- IP: <the public IP of the vm when it is provisioned and turned on>
+ IP: this doesn't matter, our module will populate this field
Type: virtualmachine
Provisioning Engine: VMware ESX
Computer Groups: allComputers
-For each virtual machine, manually update the private IP, eth0macaddress (private), and eth1macaddress (public) for the placeholder virtual machine entries created in in the computers table during step (4)
-
-Add static entries in dhcpd.conf for the eth0macaddresses (private) for all computer placeholder entries
-
-Add similar entries for each compter placeholder to the /etc/hosts file to provide VCL with name resolution on these vms
-
-Finally, once all virtual machines placeholders have been created, 'assign' them to hypervisors by using the 'Virtual Hosts' area of your web interface.
+2) 'assign' these computer placeholders (and their hostnames) to hypervisors by using the 'Virtual Hosts' area of your web interface.
-Now your hypervisors and virtual machines are setup, time to add an image:
+Now your hypervisors and virtual machines are setup. Add an image:
1) Create an entry in the image table.
name: esx3-<what you want>-v0
OSid: link to the fc9 entry on the OS table
2) Create an image revision table with the imageid fromt the entry created in (1) and the imagename used in (1)
-3) Add the image revision to the allImages and allVMimages groups
\ No newline at end of file
+3) Add the image revision to the allImages and allVMimages groups
Modified: incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.pm
URL: http://svn.apache.org/viewvc/incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.pm?rev=739839&r1=739838&r2=739839&view=diff
==============================================================================
--- incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.pm (original)
+++ incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.pm Sun Feb 1 21:12:28 2009
@@ -60,6 +60,12 @@
use VCL::utils;
use Fcntl qw(:DEFAULT :flock);
+# Used to query for the MAC address once a host has been registered
+use VMware::VIRuntime;
+use VMware::VILib;
+
+
+
##############################################################################
=head1 CLASS ATTRIBUTES
@@ -150,9 +156,10 @@
my $vmclient_OSname = $self->data->get_image_os_name;
#eventually get these from a config file or database
- my $vmhost_username = "";
- my $vmhost_password = "";
- my $datastore_ip = "";
+
+ my $vmhost_username = "root";
+ my $vmhost_password = "M\$r\@geCl0ud";
+ my $datastore_ip = "152.14.17.78";
notify($ERRORS{'OK'}, 0, "Entered ESX module, loading $image_name on $computer_shortname (on $vmhost_hostname) for reservation $reservation_id");
@@ -266,9 +273,9 @@
push(@vmxfile, "ethernet0.wakeOnPcktRcv = \"false\"\n");
push(@vmxfile, "ethernet1.wakeOnPcktRcv = \"false\"\n");
- push(@vmxfile, "ethernet0.address = \"$vmclient_eth0MAC\"\n");
- push(@vmxfile, "ethernet1.address = \"$vmclient_eth1MAC\"\n");
- push(@vmxfile, "ethernet0.addressType = \"static\"\n");
+ #push(@vmxfile, "ethernet0.address = \"$vmclient_eth0MAC\"\n");
+ #push(@vmxfile, "ethernet1.address = \"$vmclient_eth1MAC\"\n");
+ push(@vmxfile, "ethernet0.addressType = \"generated\"\n");
push(@vmxfile, "ethernet1.addressType = \"generated\"\n");
push(@vmxfile, "gui.exitOnCLIHLT = \"FALSE\"\n");
push(@vmxfile, "snapshot.disabled = \"TRUE\"\n");
@@ -324,11 +331,71 @@
notify($ERRORS{'DEBUG'},0,"Powered on: $poweron_output");
- # Start waiting for SSH to come up
+ # Query the VI Perl toolkit for the mac address of our newly registered
+ # machine
+ Vim::login(service_url => "https://$vmhost_hostname/sdk", user_name => $vmhost_username, password => $vmhost_password);
+ my $vm_view = Vim::find_entity_view(view_type => 'VirtualMachine', filter => {'config.name' => "$computer_shortname"});
+ if (!$vm_view) {
+ notify($ERRORS{'CRITICAL'}, 0, "Could not query for VM in VI PERL API");
+ Vim::logout();
+ return 0;
+ }
+ my $devices = $vm_view->config->hardware->device;
+ my $mac_addr;
+ foreach my $dev (@$devices) {
+ next unless ($dev->isa ("VirtualEthernetCard"));
+ notify($ERRORS{'DEBUG'}, 0, "deviceinfo->summary: $dev->deviceinfo->summary");
+ notify($ERRORS{'DEBUG'}, 0, "virtualswitch0: $virtualswitch0");
+ if ($dev->deviceInfo->summary eq $virtualswitch0) {
+ $mac_addr = $dev->macAddress;
+ }
+ }
+ Vim::logout();
+ if (!$mac_addr) {
+ notify($ERRORS{'CRITICAL'}, 0, "Failed to find MAC address");
+ return 0;
+ }
+ notify($ERRORS{'OK'}, 0, "Queried MAC address is $mac_addr");
+
+ # Query ARP table for $mac_addr to find the IP (waiting for machine to come up if necessary)
+ # The DHCP negotiation should add the appropriate ARP entry for us
+ my $arpstatus = 0;
+ my $wait_loops = 0;
+ my $client_ip;
+ while (!$arpstatus) {
+ my $arpoutput = `arp -n`;
+ if ($arpoutput =~ /^(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}).*?$mac_addr/mi) {
+ $client_ip = $1;
+ $arpstatus = 1;
+ notify($ERRORS{'OK'}, 0, "$computer_shortname now has ip $client_ip");
+ }
+ else {
+ if ($wait_loops > 24) {
+ notify($ERRORS{'CRITICAL'}, 0, "waited acceptable amount of time for dhcp, please check $computer_shortname on $vmhost_hostname");
+ return 0;
+ }
+ else {
+ $wait_loops++;
+ notify($ERRORS{'OK'}, 0, "going to sleep 5 seconds, waiting for computer to DHCP. Try $wait_loops");
+ sleep 5;
+ }
+ }
+ }
+
+
+ notify($ERRORS{'OK'}, 0, "Found IP address $client_ip");
+
+ # Delete existing entry for $computer_shortname in /etc/hosts (if any)
+ notify($ERRORS{'OK'}, 0, "Removing old hosts entry");
+ my $sedoutput = `sed -i "/.*\\b$computer_shortname\$/d" /etc/hosts`;
+ notify($ERRORS{'DEBUG'}, 0, $sedoutput);
+ # Add new entry to /etc/hosts for $computer_shortname
+ `echo -e "$client_ip\t$computer_shortname" >> /etc/hosts`;
+ # Start waiting for SSH to come up
my $sshdstatus = 0;
- my $wait_loops = 0;
+ $wait_loops = 0;
my $sshd_status = "off";
notify($ERRORS{'DEBUG'}, 0, "Waiting for ssh to come up on $computer_shortname");
while (!$sshdstatus) {
@@ -347,7 +414,7 @@
else {
$wait_loops++;
# to give post config a chance
- notify($ERRORS{'OK'}, 0, "going to sleep 5 seconds, waiting for post config to finish. Try $wait_loops");
+ notify($ERRORS{'OK'}, 0, "going to sleep 5 seconds, waiting for computer to start SSH. Try $wait_loops");
sleep 5;
}
} # else