You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@vcl.apache.org by Aaron Coburn <ac...@amherst.edu> on 2011/08/16 19:04:00 UTC
VMware provisioning module for vCenter clusters
Hello,
In our VCL implementation at Amherst College, we are using a VMware cluster for our virtual machine infrastructure. Because we wanted to allow VMs to float between physical hosts according to how VMware prefers to balance resource use (VMware calls this distributed resource scheduling), I wanted to access the cluster as a single datacenter through the vCenter interface. From VCL's perspective, this allows me to put all of my virtual computers on a single VM Host, when in reality each of those VMs is located on one of any number of physical machines that the VCL can remain blithely ignorant about.
The vSphere_SDK package, however, requires that vcld access a physical host directly, so I wrote a new provisioning module called VMware::vCenter. This module is a subclass of the vSphere_SDK package and works with version 2.2.1 of the VCL. The new module reimplements the vSphere_SDK subroutines that are incompatible with a datacenter-based connection, primarily anything that calls the VIExt::get_host_view() method or tries to copy or move virtual disks to new locations. There are a few other changes that are explained in the JIRA issue I created for this. I have included the relevant code in JIRA, but let me know if there is anything else I should do to help this along.
Best regards,
Aaron Coburn
Systems Administrator and Programmer
Academic Technology Services, Amherst College
(413) 542-5451 acoburn@amherst.edu
Re: VMware provisioning module for vCenter clusters
Posted by Sean Dilda <se...@duke.edu>.
This sounds a lot like William Lam's ghetto linked clones method
(http://communities.vmware.com/docs/DOC-9202). I've found it to be
pretty useful, but had some frightening corner cases around deleting VMs
and accidentally having parent VMs have their vmdk's deleted.
If you want to go down the linked clone route (which I really hope you
are, I'd love to be able to get rid of our custom code and move to
community code that has the same functionality), then I'd highly
recommend using the techniques in
http://www.vmware.com/support/developer/vc-sdk/linked_vms_note.pdf
This is VMware's published method of creating linked clones when you use
the APIs. We're using this in production and it works great and is very
quick. The only trick is you have to create a snapshot of your base
image before making a linked clone of it, but that's easily done.
On 8/17/11 8:52 PM, Andy Kurth wrote:
> This is wonderful. Thank You. I will try to incorporate it over the
> next few weeks. For anyone interested, the Jira issue is:
> https://issues.apache.org/jira/browse/VCL-499
>
> Also, I committed your improvements in VCL-470 and VCL-471 on 8/4.
>
> I'd suggest working off of trunk in Subversion and svn updating it
> regularly if you aren't already doing so. I've made some changes to
> the VMware code over the past few weeks. The vSphere.pm module has
> been improved to execute much faster. Rather than calling the VMware
> API every time something such as a datastore object is needed, it is
> only called once per object and this reference is then stored in the
> vSphere.pm object for later use. It will probably be easier to
> incorporate your changes because of this. All calls to get_host_view
> are funneled through a single subroutine.
>
> Also, the method of handling vmdk's has changed after being inspired
> by ideas shared by Dr. Masoud Sadjadi from Florida International
> University at the VCL bootcamp last month. Josh Thompson did some
> research and then I incorporated the findings. It is no longer
> necessary to create an entire copy of the vmdk for imaging and
> long-term reservations. All VMs are now configured to use the vmdk in
> persistent mode instead of independent-persistent or
> independent-nonpersistent mode. Before a VM is powered on a snapshot
> is now taken. This causes the VM to write changes to a delta file in
> the directory where the vmx is located and the master/golden vmdk
> isn't altered. If the snapshot fails for some reason, the VM is
> immediately deleted to prevent the possibility of it being powered on
> and altering the master image.
>
> The benefits are:
> -Full copies of the vmdk don't need to be made when a VM is loaded
> -Load time for imaging and long-term reservations is ~1-3 minutes
> -Load on the storage system is reduced
> -Storage consumption is reduced
> -Normal, short-term VMs can be shut down without losing the state
> -Images can be created from normal reservations
>
> For imaging reservations, the path within the vmx file is
> automatically changed when the snapshot is created to a file within
> the vmx directory. Running vmkfstools or calling the vSphere SDK to
> copy/move this vmdk causes a new, complete vmdk to be created
> containing the master image with the delta changed incorporated.
>
> The code to create a full independent-persistent copy is still intact.
> This mode is currently only used if a reservation is a server request
> -- a major new feature in VCL 2.3. There is supposed to be a slight
> performance benefit of using this mode so I want to keep it as an
> option.
>
> Thanks for your contributions. Would you be interested in becoming a
> committer to the project?
>
> Regards,
> Andy
>
> On Tue, Aug 16, 2011 at 1:04 PM, Aaron Coburn<ac...@amherst.edu> wrote:
>> Hello,
>> In our VCL implementation at Amherst College, we are using a VMware cluster for our virtual machine infrastructure. Because we wanted to allow VMs to float between physical hosts according to how VMware prefers to balance resource use (VMware calls this distributed resource scheduling), I wanted to access the cluster as a single datacenter through the vCenter interface. From VCL's perspective, this allows me to put all of my virtual computers on a single VM Host, when in reality each of those VMs is located on one of any number of physical machines that the VCL can remain blithely ignorant about.
>>
>> The vSphere_SDK package, however, requires that vcld access a physical host directly, so I wrote a new provisioning module called VMware::vCenter. This module is a subclass of the vSphere_SDK package and works with version 2.2.1 of the VCL. The new module reimplements the vSphere_SDK subroutines that are incompatible with a datacenter-based connection, primarily anything that calls the VIExt::get_host_view() method or tries to copy or move virtual disks to new locations. There are a few other changes that are explained in the JIRA issue I created for this. I have included the relevant code in JIRA, but let me know if there is anything else I should do to help this along.
>>
>> Best regards,
>>
>> Aaron Coburn
>> Systems Administrator and Programmer
>> Academic Technology Services, Amherst College
>> (413) 542-5451 acoburn@amherst.edu
>>
>>
>>
>>
>>
>>
Re: VMware provisioning module for vCenter clusters
Posted by Aaron Coburn <ac...@amherst.edu>.
Andy, I can certainly make the initial commit of the module.
An interesting problem I have had with this module, however, is that when I clone a VM (which is the only way to move a virtual disk using the vSphere API while preserving thin provisioning), VMWare would silently truncate the name and enclosing directory of the virtual disk -- though only if the basename of the VM is longer than 29 characters. The actual name of the VM can still be up to 80 characters, but the location of the virtual disk, it appears, cannot. And since the VCL crafts its own *.VMX files, it needs to know precisely where to find the *.VMDKs.
For example, a virtual disk with this name: vmwarewinxp-WindowsNoApps82-v3 (basename has 30 chars) would be saved in this location:
vmwarewinxp-WindowsNoApps82-v/vmwarewinxp-WindowsNoApps82-v.vmdk
As another example, vmwarewin7-WindowsNoApps52-v0 would not have a problem until it hit version 10 (basename has 29 chars), at which point it would be put in this location:
vmwarewin7-WindowsNoApps52-v1_X/vmwarewin7-WindowsNoApps52-v1_X.vmdk
(where X would be some number that VMware appends to the virtual disk if version1 is still available on the datastore -- X will increment on subsequent attempts -- note that the basename is now more than 29 total characters)
With the VCL naming conventions (ostype-imagenameID-version), it is, of course, easy to exceed the 29-character limit. I was unsuccessful in locating any documentation from VMware about this. The work-around, though, was to add a few lines in the web code to keep the value of vcl.revision.imagename limited to 29 chars. Clearly, it is necessary to retain the version suffix as well as the imageID, so the code pulls apart the imagename clauses, shortens the middle clause and reassembles them into a 29-character-or-less string. Similar code would need to be added to the "-setup" part of vcld.
As you know, this imagename value is never actually seen by a user, so I felt at some liberty in modifying how it is generated. Does that seem like a reasonable tact?
Aaron
On Jan 30, 2012, at 2:36 PM, Andy Kurth wrote:
> Hi Aaron,
> How would you like to proceed with your vCenter modules now that
> you're a committer? You can make the initial commit if you're
> comfortable with it. If not, I now have access to a vCenter license
> and can incorporate it into the current code in trunk and make the
> initial commit. There have been some minor changes to vSphere.pm so
> some changes may be necessary.
>
> Thanks,
> Andy
>
> On Fri, Aug 19, 2011 at 10:01 AM, Aaron Coburn <ac...@amherst.edu> wrote:
>> Hi, Andy,
>> thanks for the details on where the VMware module is headed. I am interested in becoming a committer for the project, though having my institution sign off on the Apache license will require a bit of, shall we say, process. I'll be in touch about that, though.
>>
>> Aaron
>>
>>
>>
>> On Aug 17, 2011, at 8:52 PM, Andy Kurth wrote:
>>
>>> This is wonderful. Thank You. I will try to incorporate it over the
>>> next few weeks. For anyone interested, the Jira issue is:
>>> https://issues.apache.org/jira/browse/VCL-499
>>>
>>> Also, I committed your improvements in VCL-470 and VCL-471 on 8/4.
>>>
>>> I'd suggest working off of trunk in Subversion and svn updating it
>>> regularly if you aren't already doing so. I've made some changes to
>>> the VMware code over the past few weeks. The vSphere.pm module has
>>> been improved to execute much faster. Rather than calling the VMware
>>> API every time something such as a datastore object is needed, it is
>>> only called once per object and this reference is then stored in the
>>> vSphere.pm object for later use. It will probably be easier to
>>> incorporate your changes because of this. All calls to get_host_view
>>> are funneled through a single subroutine.
>>>
>>> Also, the method of handling vmdk's has changed after being inspired
>>> by ideas shared by Dr. Masoud Sadjadi from Florida International
>>> University at the VCL bootcamp last month. Josh Thompson did some
>>> research and then I incorporated the findings. It is no longer
>>> necessary to create an entire copy of the vmdk for imaging and
>>> long-term reservations. All VMs are now configured to use the vmdk in
>>> persistent mode instead of independent-persistent or
>>> independent-nonpersistent mode. Before a VM is powered on a snapshot
>>> is now taken. This causes the VM to write changes to a delta file in
>>> the directory where the vmx is located and the master/golden vmdk
>>> isn't altered. If the snapshot fails for some reason, the VM is
>>> immediately deleted to prevent the possibility of it being powered on
>>> and altering the master image.
>>>
>>> The benefits are:
>>> -Full copies of the vmdk don't need to be made when a VM is loaded
>>> -Load time for imaging and long-term reservations is ~1-3 minutes
>>> -Load on the storage system is reduced
>>> -Storage consumption is reduced
>>> -Normal, short-term VMs can be shut down without losing the state
>>> -Images can be created from normal reservations
>>>
>>> For imaging reservations, the path within the vmx file is
>>> automatically changed when the snapshot is created to a file within
>>> the vmx directory. Running vmkfstools or calling the vSphere SDK to
>>> copy/move this vmdk causes a new, complete vmdk to be created
>>> containing the master image with the delta changed incorporated.
>>>
>>> The code to create a full independent-persistent copy is still intact.
>>> This mode is currently only used if a reservation is a server request
>>> -- a major new feature in VCL 2.3. There is supposed to be a slight
>>> performance benefit of using this mode so I want to keep it as an
>>> option.
>>>
>>> Thanks for your contributions. Would you be interested in becoming a
>>> committer to the project?
>>>
>>> Regards,
>>> Andy
>>>
>>> On Tue, Aug 16, 2011 at 1:04 PM, Aaron Coburn <ac...@amherst.edu> wrote:
>>>> Hello,
>>>> In our VCL implementation at Amherst College, we are using a VMware cluster for our virtual machine infrastructure. Because we wanted to allow VMs to float between physical hosts according to how VMware prefers to balance resource use (VMware calls this distributed resource scheduling), I wanted to access the cluster as a single datacenter through the vCenter interface. From VCL's perspective, this allows me to put all of my virtual computers on a single VM Host, when in reality each of those VMs is located on one of any number of physical machines that the VCL can remain blithely ignorant about.
>>>>
>>>> The vSphere_SDK package, however, requires that vcld access a physical host directly, so I wrote a new provisioning module called VMware::vCenter. This module is a subclass of the vSphere_SDK package and works with version 2.2.1 of the VCL. The new module reimplements the vSphere_SDK subroutines that are incompatible with a datacenter-based connection, primarily anything that calls the VIExt::get_host_view() method or tries to copy or move virtual disks to new locations. There are a few other changes that are explained in the JIRA issue I created for this. I have included the relevant code in JIRA, but let me know if there is anything else I should do to help this along.
>>>>
>>>> Best regards,
>>>>
>>>> Aaron Coburn
>>>> Systems Administrator and Programmer
>>>> Academic Technology Services, Amherst College
>>>> (413) 542-5451 acoburn@amherst.edu
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>
Re: VMware provisioning module for vCenter clusters
Posted by Andy Kurth <an...@ncsu.edu>.
Hi Aaron,
How would you like to proceed with your vCenter modules now that
you're a committer? You can make the initial commit if you're
comfortable with it. If not, I now have access to a vCenter license
and can incorporate it into the current code in trunk and make the
initial commit. There have been some minor changes to vSphere.pm so
some changes may be necessary.
Thanks,
Andy
On Fri, Aug 19, 2011 at 10:01 AM, Aaron Coburn <ac...@amherst.edu> wrote:
> Hi, Andy,
> thanks for the details on where the VMware module is headed. I am interested in becoming a committer for the project, though having my institution sign off on the Apache license will require a bit of, shall we say, process. I'll be in touch about that, though.
>
> Aaron
>
>
>
> On Aug 17, 2011, at 8:52 PM, Andy Kurth wrote:
>
>> This is wonderful. Thank You. I will try to incorporate it over the
>> next few weeks. For anyone interested, the Jira issue is:
>> https://issues.apache.org/jira/browse/VCL-499
>>
>> Also, I committed your improvements in VCL-470 and VCL-471 on 8/4.
>>
>> I'd suggest working off of trunk in Subversion and svn updating it
>> regularly if you aren't already doing so. I've made some changes to
>> the VMware code over the past few weeks. The vSphere.pm module has
>> been improved to execute much faster. Rather than calling the VMware
>> API every time something such as a datastore object is needed, it is
>> only called once per object and this reference is then stored in the
>> vSphere.pm object for later use. It will probably be easier to
>> incorporate your changes because of this. All calls to get_host_view
>> are funneled through a single subroutine.
>>
>> Also, the method of handling vmdk's has changed after being inspired
>> by ideas shared by Dr. Masoud Sadjadi from Florida International
>> University at the VCL bootcamp last month. Josh Thompson did some
>> research and then I incorporated the findings. It is no longer
>> necessary to create an entire copy of the vmdk for imaging and
>> long-term reservations. All VMs are now configured to use the vmdk in
>> persistent mode instead of independent-persistent or
>> independent-nonpersistent mode. Before a VM is powered on a snapshot
>> is now taken. This causes the VM to write changes to a delta file in
>> the directory where the vmx is located and the master/golden vmdk
>> isn't altered. If the snapshot fails for some reason, the VM is
>> immediately deleted to prevent the possibility of it being powered on
>> and altering the master image.
>>
>> The benefits are:
>> -Full copies of the vmdk don't need to be made when a VM is loaded
>> -Load time for imaging and long-term reservations is ~1-3 minutes
>> -Load on the storage system is reduced
>> -Storage consumption is reduced
>> -Normal, short-term VMs can be shut down without losing the state
>> -Images can be created from normal reservations
>>
>> For imaging reservations, the path within the vmx file is
>> automatically changed when the snapshot is created to a file within
>> the vmx directory. Running vmkfstools or calling the vSphere SDK to
>> copy/move this vmdk causes a new, complete vmdk to be created
>> containing the master image with the delta changed incorporated.
>>
>> The code to create a full independent-persistent copy is still intact.
>> This mode is currently only used if a reservation is a server request
>> -- a major new feature in VCL 2.3. There is supposed to be a slight
>> performance benefit of using this mode so I want to keep it as an
>> option.
>>
>> Thanks for your contributions. Would you be interested in becoming a
>> committer to the project?
>>
>> Regards,
>> Andy
>>
>> On Tue, Aug 16, 2011 at 1:04 PM, Aaron Coburn <ac...@amherst.edu> wrote:
>>> Hello,
>>> In our VCL implementation at Amherst College, we are using a VMware cluster for our virtual machine infrastructure. Because we wanted to allow VMs to float between physical hosts according to how VMware prefers to balance resource use (VMware calls this distributed resource scheduling), I wanted to access the cluster as a single datacenter through the vCenter interface. From VCL's perspective, this allows me to put all of my virtual computers on a single VM Host, when in reality each of those VMs is located on one of any number of physical machines that the VCL can remain blithely ignorant about.
>>>
>>> The vSphere_SDK package, however, requires that vcld access a physical host directly, so I wrote a new provisioning module called VMware::vCenter. This module is a subclass of the vSphere_SDK package and works with version 2.2.1 of the VCL. The new module reimplements the vSphere_SDK subroutines that are incompatible with a datacenter-based connection, primarily anything that calls the VIExt::get_host_view() method or tries to copy or move virtual disks to new locations. There are a few other changes that are explained in the JIRA issue I created for this. I have included the relevant code in JIRA, but let me know if there is anything else I should do to help this along.
>>>
>>> Best regards,
>>>
>>> Aaron Coburn
>>> Systems Administrator and Programmer
>>> Academic Technology Services, Amherst College
>>> (413) 542-5451 acoburn@amherst.edu
>>>
>>>
>>>
>>>
>>>
>>>
>
Re: VMware provisioning module for vCenter clusters
Posted by Andy Kurth <an...@ncsu.edu>.
This is wonderful. Thank You. I will try to incorporate it over the
next few weeks. For anyone interested, the Jira issue is:
https://issues.apache.org/jira/browse/VCL-499
Also, I committed your improvements in VCL-470 and VCL-471 on 8/4.
I'd suggest working off of trunk in Subversion and svn updating it
regularly if you aren't already doing so. I've made some changes to
the VMware code over the past few weeks. The vSphere.pm module has
been improved to execute much faster. Rather than calling the VMware
API every time something such as a datastore object is needed, it is
only called once per object and this reference is then stored in the
vSphere.pm object for later use. It will probably be easier to
incorporate your changes because of this. All calls to get_host_view
are funneled through a single subroutine.
Also, the method of handling vmdk's has changed after being inspired
by ideas shared by Dr. Masoud Sadjadi from Florida International
University at the VCL bootcamp last month. Josh Thompson did some
research and then I incorporated the findings. It is no longer
necessary to create an entire copy of the vmdk for imaging and
long-term reservations. All VMs are now configured to use the vmdk in
persistent mode instead of independent-persistent or
independent-nonpersistent mode. Before a VM is powered on a snapshot
is now taken. This causes the VM to write changes to a delta file in
the directory where the vmx is located and the master/golden vmdk
isn't altered. If the snapshot fails for some reason, the VM is
immediately deleted to prevent the possibility of it being powered on
and altering the master image.
The benefits are:
-Full copies of the vmdk don't need to be made when a VM is loaded
-Load time for imaging and long-term reservations is ~1-3 minutes
-Load on the storage system is reduced
-Storage consumption is reduced
-Normal, short-term VMs can be shut down without losing the state
-Images can be created from normal reservations
For imaging reservations, the path within the vmx file is
automatically changed when the snapshot is created to a file within
the vmx directory. Running vmkfstools or calling the vSphere SDK to
copy/move this vmdk causes a new, complete vmdk to be created
containing the master image with the delta changed incorporated.
The code to create a full independent-persistent copy is still intact.
This mode is currently only used if a reservation is a server request
-- a major new feature in VCL 2.3. There is supposed to be a slight
performance benefit of using this mode so I want to keep it as an
option.
Thanks for your contributions. Would you be interested in becoming a
committer to the project?
Regards,
Andy
On Tue, Aug 16, 2011 at 1:04 PM, Aaron Coburn <ac...@amherst.edu> wrote:
> Hello,
> In our VCL implementation at Amherst College, we are using a VMware cluster for our virtual machine infrastructure. Because we wanted to allow VMs to float between physical hosts according to how VMware prefers to balance resource use (VMware calls this distributed resource scheduling), I wanted to access the cluster as a single datacenter through the vCenter interface. >From VCL's perspective, this allows me to put all of my virtual computers on a single VM Host, when in reality each of those VMs is located on one of any number of physical machines that the VCL can remain blithely ignorant about.
>
> The vSphere_SDK package, however, requires that vcld access a physical host directly, so I wrote a new provisioning module called VMware::vCenter. This module is a subclass of the vSphere_SDK package and works with version 2.2.1 of the VCL. The new module reimplements the vSphere_SDK subroutines that are incompatible with a datacenter-based connection, primarily anything that calls the VIExt::get_host_view() method or tries to copy or move virtual disks to new locations. There are a few other changes that are explained in the JIRA issue I created for this. I have included the relevant code in JIRA, but let me know if there is anything else I should do to help this along.
>
> Best regards,
>
> Aaron Coburn
> Systems Administrator and Programmer
> Academic Technology Services, Amherst College
> (413) 542-5451 acoburn@amherst.edu
>
>
>
>
>
>
Re: VMware provisioning module for vCenter clusters
Posted by Aaron Peeler <aa...@ncsu.edu>.
Hi Aaron,
Thanks. This is great, definitely needed a method on provisioning on vcenter.
I think Andy will followup as soon as he can on this. He has the most
knowledge of the current committers on the VMware module framework.
Aaron
On Tue, Aug 16, 2011 at 1:04 PM, Aaron Coburn <ac...@amherst.edu> wrote:
> Hello,
> In our VCL implementation at Amherst College, we are using a VMware cluster for our virtual machine infrastructure. Because we wanted to allow VMs to float between physical hosts according to how VMware prefers to balance resource use (VMware calls this distributed resource scheduling), I wanted to access the cluster as a single datacenter through the vCenter interface. >From VCL's perspective, this allows me to put all of my virtual computers on a single VM Host, when in reality each of those VMs is located on one of any number of physical machines that the VCL can remain blithely ignorant about.
>
> The vSphere_SDK package, however, requires that vcld access a physical host directly, so I wrote a new provisioning module called VMware::vCenter. This module is a subclass of the vSphere_SDK package and works with version 2.2.1 of the VCL. The new module reimplements the vSphere_SDK subroutines that are incompatible with a datacenter-based connection, primarily anything that calls the VIExt::get_host_view() method or tries to copy or move virtual disks to new locations. There are a few other changes that are explained in the JIRA issue I created for this. I have included the relevant code in JIRA, but let me know if there is anything else I should do to help this along.
>
> Best regards,
>
> Aaron Coburn
> Systems Administrator and Programmer
> Academic Technology Services, Amherst College
> (413) 542-5451 acoburn@amherst.edu
>
>
>
>
>
>
--
Aaron Peeler
Program Manager
Virtual Computing Lab
NC State University
All electronic mail messages in connection with State business which
are sent to or received by this account are subject to the NC Public
Records Law and may be disclosed to third parties.