You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@vcl.apache.org by Josh Thompson <jo...@ncsu.edu> on 2012/04/25 16:28:30 UTC
status of 2.3 release
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
I think we targeted April to get a release out. The number of JIRA issues we
have left is starting to go down a fair amount. I'm almost finished with the
frontend. How's the backend? Are there other things we'd like to do for this
release such as scripting the install as much as possible? I don't think
we'll be done by the end of April, but I think the first week of May would be
possible.
Aaron C., Jim, and David - Have you tested your provisioning modules with
recent code from subversion? If not, do you think you can do that in the next
week or two?
Josh
- --
- -------------------------------
Josh Thompson
VCL Developer
North Carolina State University
my GPG/PGP key can be found at pgp.mit.edu
All electronic mail messages in connection with State business which
are sent to or received by this account are subject to the NC Public
Records Law and may be disclosed to third parties.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
iEYEARECAAYFAk+YChIACgkQV/LQcNdtPQPsagCdHCZOuT1ZIXxZrB+9EtppUjkc
KJIAn1mxEVtuZ64KEG0I4AwosUo3qT2P
=InjJ
-----END PGP SIGNATURE-----
Re: status of 2.3 release
Posted by Andy Kurth <an...@ncsu.edu>.
I agree that an overall naming convention shouldn't be imposed because
of one particular provisioning engine. Your idea of having
copy/move_virtual_disk return the name of the destination disk sounds
like the best approach. I wouldn't have these subroutines edit the
image name in the database, instead the capture subroutine should do
this.
I'd move the logic you had in the web code to copy/move_virtual_disk
after the "logical" method failed and before the clone procedure
starts so that the cloned VM gets the shortened name. As a safety
check, once you construct the new name you can call
get_image_info($new_image_name). If this returns null then you know
there isn't a naming conflict. It will display a warning but a flag
can be added to get_image_info to prevent this.
There is a new update_image_name subroutine in utils.pm which updates
both image.name and imagerevision.name. The code in capture() would
look something like this:
my $new_image_name = $self->api->copy_virtual_disk(..., ...);
if ($new_image_name ne $image_name) {
update_image_name($image_id, $imagerevision_id, $new_image_name);
$self->data->set_image_name($new_image_name);
}
You don't have to call $self->data->refresh. This would only be
necessary in cases where another process may have updated something in
the database and you want to make sure you have the current data.
Thanks for helping out with this,
Andy
On Thu, Apr 26, 2012 at 12:06 PM, Aaron Coburn <ac...@amherst.edu> wrote:
> Hi, Andy,
>
> Please feel free to make the changes you described in 1 and 2 to trunk.
>
>
> Will do.
>
> Number 3 is a bit trickier. I'm afraid setting a name length limit in
> the database will break other things.
>
>
> Is the directory name also truncated or left intact? Example:
> /vmfs/volumes/datastore/vmwarewin2008-myreallylongname-v0/vmwarewin2008-myreallylongna.vmdk
> -or-
> /vmfs/volumes/datastore/vmwarewin2008-myreallylongna/vmwarewin2008-myreallylongna.vmdk
>
>
> Yes, the directory names are also truncated. And I am aware of the issues
> you listed below.
>
> The way I handled this was more involved that simply modifying the db schema
> (which I actually didn't touch for this). The snippet of code below is from
> my .ht-inc/image.php:
>
> $vmware_max_len = 29;
> if(strlen($newname) > $vmware_max_len){
> $matches = array();
> if(preg_match("/^(\w+)-(\w+?)(\d*)-(v\d+)$/", $newname, $matches)){
> list($base, $name, $imgid, $version) = array_splice($matches,
> 1);
> $shortened = substr($name, 0, $vmware_max_len - 2 -
> strlen($imgid) - strlen($base) - strlen($version));
> $newname = $base . "-" . $shortened . $imgid . "-" . $version;
> } else {
> preg_match("/^(.{15}).*(.{10})$/", $newname, $matches);
> $newname = $matches[1] . $matches[2];
> }
> }
>
> Effectively, if the $newname is longer than 29 characters, I split up the
> name into four sections (base os, image name, image id and image version). I
> then keep the base os, image name and image version as-is and truncate the
> image name as needed. This effectively solves the issue of image name
> uniqueness. (If the code isn't able to break the imagename up into a
> quartet, I use a more brute-force approach but still retain the parts of the
> path that ensure uniqueness.)
>
> What I don't like about this approach is that it enforces a VCL-wide naming
> convention based on an undocumented "feature" of one particular provisioning
> engine. There had been some discussion earlier on this list about letting
> VMware manage its own image names and simply having the VCL follow whatever
> VMware chooses to do. I like that idea in principle, but the reality is that
> the VCL *is* managing the names of virtual disks. Given that structure, I
> preferred the approach described above over letting VMware manage its own
> names.
> -Aaron
>
> --
> Aaron Coburn
> Systems Administrator and Programmer
> Academic Technology Services, Amherst College
> acoburn@amherst.edu
>
>
>
>
> Assuming the directory name is NOT truncated, what do you think about
>
> leaving the name in the database as-is
> (vmwarewin2008-myreallylongname-v0) even if the vmdk file name gets
> truncated. Add additional code to get the actual vmdk file name
> rather than constructing it. This would probably be done in
> VMware.pm::does_image_exist. Rather than looking for the full,
> non-truncated vmdk file path it could get a list of vmdk directory
> contents and then fetch the actual file name. Once retrieved,
> set_vmdk_file_path could be called. All of the get_vmdk* subroutines
> should then automatically use the correct value.
>
> If the directory name gets truncated then there could be additional
> problems if the truncated winds up being identical to another existing
> name. Example:
> vmwarewin2008-myreallylongname-v10
> -gets truncated to-
> vmwarewin2008-myreallylongname-v1
>
> This would cause conflicts in both the filesystem and database. The
> database wouldn't allow you to change the image name because
> image.name has a unique restriction.
>
> -Andy
>
>
> On Wed, Apr 25, 2012 at 12:58 PM, Aaron Coburn <ac...@amherst.edu> wrote:
>
> I looked through the current code for
> VCL::Module::Provisioning::VMware::vSphere_SDK
>
>
> For the most part, all of the code I submitted for the vCenter module has
> been integrated into vSphere_SDK class.
>
>
> There are three exceptions, which I didn't see in the current code:
>
>
> 1. in the move_virtual_disk subroutine, you use the MoveVirtualDisk vSphere
> method. This works fine on esx hosts, but in the context of a vCenter, it
> will return a "not implemented" fault. I liked how this was handled in the
> copy_virtual_disk subroutine: first try the method that makes sense and then
> fall back to using the full 'CloneVM' approach if the first method doesn't
> work. The way I had implemented this, move_virtual_disk just uses the
> copy_virtual_disk method and then deletes the source files upon successful
> completion. I can add that code if you would like.
>
>
> 2. in the copy_file_to and copy_file_from subroutines, I needed to add a
> loop because the VIExt::http_put_file and VIExt::http_get_file functions
> periodically fail. That is, if copy_file_to fails, the code sleeps for a few
> seconds and tries again, up to a maximum of a few tries. From what I can
> tell, this sort of looping doesn't exist when prepare_vmx() is called. I can
> wrap that in a Module::code_loop_timeout subroutine, if you would like.
>
>
> 3. When vCenter clones a VM with the CloneVM method, it will silently
> truncate long vm names. This isn't necessarily a problem as long as that new
> name finds its way into the database; otherwise, however, when VMX files are
> generated, the vmdk file pointers will reference incorrect locations. I had
> solved this by enforcing a maximum length in the web code (i.e. the
> vcl.image.name table). There are pros and cons to this, though a better
> approach may be to retrieve the actual name of the vmdk file after a clone
> operation is performed. This would require having the copy_virtual_disk and
> move_virtual_disk to return values other than just 1 (or, those methods
> could update the database themselves). I am not sure which approach you
> would prefer. I am also unsure of the best way to update the database with
> this information: would I simply call database_execute(...) and then call
> $self->data->refresh()? Or is there other code that handles this type of
> operation? Please advise.
>
>
> Aaron
>
>
>
> On Apr 25, 2012, at 10:28 AM, Josh Thompson wrote:
>
>
> -----BEGIN PGP SIGNED MESSAGE-----
>
> Hash: SHA1
>
>
> I think we targeted April to get a release out. The number of JIRA issues
> we
>
> have left is starting to go down a fair amount. I'm almost finished with
> the
>
> frontend. How's the backend? Are there other things we'd like to do for
> this
>
> release such as scripting the install as much as possible? I don't think
>
> we'll be done by the end of April, but I think the first week of May would
> be
>
> possible.
>
>
> Aaron C., Jim, and David - Have you tested your provisioning modules with
>
> recent code from subversion? If not, do you think you can do that in the
> next
>
> week or two?
>
>
> Josh
>
> - --
>
> - -------------------------------
>
> Josh Thompson
>
> VCL Developer
>
> North Carolina State University
>
>
> my GPG/PGP key can be found at pgp.mit.edu
>
>
> All electronic mail messages in connection with State business which
>
> are sent to or received by this account are subject to the NC Public
>
> Records Law and may be disclosed to third parties.
>
> -----BEGIN PGP SIGNATURE-----
>
> Version: GnuPG v2.0.17 (GNU/Linux)
>
>
> iEYEARECAAYFAk+YChIACgkQV/LQcNdtPQPsagCdHCZOuT1ZIXxZrB+9EtppUjkc
>
> KJIAn1mxEVtuZ64KEG0I4AwosUo3qT2P
>
> =InjJ
>
> -----END PGP SIGNATURE-----
>
>
>
>
Re: status of 2.3 release
Posted by Aaron Coburn <ac...@amherst.edu>.
Hi, Andy,
> Please feel free to make the changes you described in 1 and 2 to trunk.
Will do.
> Number 3 is a bit trickier. I'm afraid setting a name length limit in
> the database will break other things.
>
> Is the directory name also truncated or left intact? Example:
> /vmfs/volumes/datastore/vmwarewin2008-myreallylongname-v0/vmwarewin2008-myreallylongna.vmdk
> -or-
> /vmfs/volumes/datastore/vmwarewin2008-myreallylongna/vmwarewin2008-myreallylongna.vmdk
Yes, the directory names are also truncated. And I am aware of the issues you listed below.
The way I handled this was more involved that simply modifying the db schema (which I actually didn't touch for this). The snippet of code below is from my .ht-inc/image.php:
$vmware_max_len = 29;
if(strlen($newname) > $vmware_max_len){
$matches = array();
if(preg_match("/^(\w+)-(\w+?)(\d*)-(v\d+)$/", $newname, $matches)){
list($base, $name, $imgid, $version) = array_splice($matches, 1);
$shortened = substr($name, 0, $vmware_max_len - 2 - strlen($imgid) - strlen($base) - strlen($version));
$newname = $base . "-" . $shortened . $imgid . "-" . $version;
} else {
preg_match("/^(.{15}).*(.{10})$/", $newname, $matches);
$newname = $matches[1] . $matches[2];
}
}
Effectively, if the $newname is longer than 29 characters, I split up the name into four sections (base os, image name, image id and image version). I then keep the base os, image name and image version as-is and truncate the image name as needed. This effectively solves the issue of image name uniqueness. (If the code isn't able to break the imagename up into a quartet, I use a more brute-force approach but still retain the parts of the path that ensure uniqueness.)
What I don't like about this approach is that it enforces a VCL-wide naming convention based on an undocumented "feature" of one particular provisioning engine. There had been some discussion earlier on this list about letting VMware manage its own image names and simply having the VCL follow whatever VMware chooses to do. I like that idea in principle, but the reality is that the VCL *is* managing the names of virtual disks. Given that structure, I preferred the approach described above over letting VMware manage its own names.
-Aaron
--
Aaron Coburn
Systems Administrator and Programmer
Academic Technology Services, Amherst College
acoburn@amherst.edu
> Assuming the directory name is NOT truncated, what do you think about
> leaving the name in the database as-is
> (vmwarewin2008-myreallylongname-v0) even if the vmdk file name gets
> truncated. Add additional code to get the actual vmdk file name
> rather than constructing it. This would probably be done in
> VMware.pm::does_image_exist. Rather than looking for the full,
> non-truncated vmdk file path it could get a list of vmdk directory
> contents and then fetch the actual file name. Once retrieved,
> set_vmdk_file_path could be called. All of the get_vmdk* subroutines
> should then automatically use the correct value.
>
> If the directory name gets truncated then there could be additional
> problems if the truncated winds up being identical to another existing
> name. Example:
> vmwarewin2008-myreallylongname-v10
> -gets truncated to-
> vmwarewin2008-myreallylongname-v1
>
> This would cause conflicts in both the filesystem and database. The
> database wouldn't allow you to change the image name because
> image.name has a unique restriction.
>
> -Andy
>
>
> On Wed, Apr 25, 2012 at 12:58 PM, Aaron Coburn <ac...@amherst.edu> wrote:
>> I looked through the current code for VCL::Module::Provisioning::VMware::vSphere_SDK
>>
>> For the most part, all of the code I submitted for the vCenter module has been integrated into vSphere_SDK class.
>>
>> There are three exceptions, which I didn't see in the current code:
>>
>> 1. in the move_virtual_disk subroutine, you use the MoveVirtualDisk vSphere method. This works fine on esx hosts, but in the context of a vCenter, it will return a "not implemented" fault. I liked how this was handled in the copy_virtual_disk subroutine: first try the method that makes sense and then fall back to using the full 'CloneVM' approach if the first method doesn't work. The way I had implemented this, move_virtual_disk just uses the copy_virtual_disk method and then deletes the source files upon successful completion. I can add that code if you would like.
>>
>> 2. in the copy_file_to and copy_file_from subroutines, I needed to add a loop because the VIExt::http_put_file and VIExt::http_get_file functions periodically fail. That is, if copy_file_to fails, the code sleeps for a few seconds and tries again, up to a maximum of a few tries. From what I can tell, this sort of looping doesn't exist when prepare_vmx() is called. I can wrap that in a Module::code_loop_timeout subroutine, if you would like.
>>
>> 3. When vCenter clones a VM with the CloneVM method, it will silently truncate long vm names. This isn't necessarily a problem as long as that new name finds its way into the database; otherwise, however, when VMX files are generated, the vmdk file pointers will reference incorrect locations. I had solved this by enforcing a maximum length in the web code (i.e. the vcl.image.name table). There are pros and cons to this, though a better approach may be to retrieve the actual name of the vmdk file after a clone operation is performed. This would require having the copy_virtual_disk and move_virtual_disk to return values other than just 1 (or, those methods could update the database themselves). I am not sure which approach you would prefer. I am also unsure of the best way to update the database with this information: would I simply call database_execute(...) and then call $self->data->refresh()? Or is there other code that handles this type of operation? Please advise.
>>
>> Aaron
>>
>>
>> On Apr 25, 2012, at 10:28 AM, Josh Thompson wrote:
>>
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA1
>>>
>>> I think we targeted April to get a release out. The number of JIRA issues we
>>> have left is starting to go down a fair amount. I'm almost finished with the
>>> frontend. How's the backend? Are there other things we'd like to do for this
>>> release such as scripting the install as much as possible? I don't think
>>> we'll be done by the end of April, but I think the first week of May would be
>>> possible.
>>>
>>> Aaron C., Jim, and David - Have you tested your provisioning modules with
>>> recent code from subversion? If not, do you think you can do that in the next
>>> week or two?
>>>
>>> Josh
>>> - --
>>> - -------------------------------
>>> Josh Thompson
>>> VCL Developer
>>> North Carolina State University
>>>
>>> my GPG/PGP key can be found at pgp.mit.edu
>>>
>>> All electronic mail messages in connection with State business which
>>> are sent to or received by this account are subject to the NC Public
>>> Records Law and may be disclosed to third parties.
>>> -----BEGIN PGP SIGNATURE-----
>>> Version: GnuPG v2.0.17 (GNU/Linux)
>>>
>>> iEYEARECAAYFAk+YChIACgkQV/LQcNdtPQPsagCdHCZOuT1ZIXxZrB+9EtppUjkc
>>> KJIAn1mxEVtuZ64KEG0I4AwosUo3qT2P
>>> =InjJ
>>> -----END PGP SIGNATURE-----
>>>
>>
Re: status of 2.3 release
Posted by Andy Kurth <an...@ncsu.edu>.
Hi Aaron,
Please feel free to make the changes you described in 1 and 2 to trunk.
Number 3 is a bit trickier. I'm afraid setting a name length limit in
the database will break other things.
Is the directory name also truncated or left intact? Example:
/vmfs/volumes/datastore/vmwarewin2008-myreallylongname-v0/vmwarewin2008-myreallylongna.vmdk
-or-
/vmfs/volumes/datastore/vmwarewin2008-myreallylongna/vmwarewin2008-myreallylongna.vmdk
Assuming the directory name is NOT truncated, what do you think about
leaving the name in the database as-is
(vmwarewin2008-myreallylongname-v0) even if the vmdk file name gets
truncated. Add additional code to get the actual vmdk file name
rather than constructing it. This would probably be done in
VMware.pm::does_image_exist. Rather than looking for the full,
non-truncated vmdk file path it could get a list of vmdk directory
contents and then fetch the actual file name. Once retrieved,
set_vmdk_file_path could be called. All of the get_vmdk* subroutines
should then automatically use the correct value.
If the directory name gets truncated then there could be additional
problems if the truncated winds up being identical to another existing
name. Example:
vmwarewin2008-myreallylongname-v10
-gets truncated to-
vmwarewin2008-myreallylongname-v1
This would cause conflicts in both the filesystem and database. The
database wouldn't allow you to change the image name because
image.name has a unique restriction.
-Andy
On Wed, Apr 25, 2012 at 12:58 PM, Aaron Coburn <ac...@amherst.edu> wrote:
> I looked through the current code for VCL::Module::Provisioning::VMware::vSphere_SDK
>
> For the most part, all of the code I submitted for the vCenter module has been integrated into vSphere_SDK class.
>
> There are three exceptions, which I didn't see in the current code:
>
> 1. in the move_virtual_disk subroutine, you use the MoveVirtualDisk vSphere method. This works fine on esx hosts, but in the context of a vCenter, it will return a "not implemented" fault. I liked how this was handled in the copy_virtual_disk subroutine: first try the method that makes sense and then fall back to using the full 'CloneVM' approach if the first method doesn't work. The way I had implemented this, move_virtual_disk just uses the copy_virtual_disk method and then deletes the source files upon successful completion. I can add that code if you would like.
>
> 2. in the copy_file_to and copy_file_from subroutines, I needed to add a loop because the VIExt::http_put_file and VIExt::http_get_file functions periodically fail. That is, if copy_file_to fails, the code sleeps for a few seconds and tries again, up to a maximum of a few tries. From what I can tell, this sort of looping doesn't exist when prepare_vmx() is called. I can wrap that in a Module::code_loop_timeout subroutine, if you would like.
>
> 3. When vCenter clones a VM with the CloneVM method, it will silently truncate long vm names. This isn't necessarily a problem as long as that new name finds its way into the database; otherwise, however, when VMX files are generated, the vmdk file pointers will reference incorrect locations. I had solved this by enforcing a maximum length in the web code (i.e. the vcl.image.name table). There are pros and cons to this, though a better approach may be to retrieve the actual name of the vmdk file after a clone operation is performed. This would require having the copy_virtual_disk and move_virtual_disk to return values other than just 1 (or, those methods could update the database themselves). I am not sure which approach you would prefer. I am also unsure of the best way to update the database with this information: would I simply call database_execute(...) and then call $self->data->refresh()? Or is there other code that handles this type of operation? Please advise.
>
> Aaron
>
>
> On Apr 25, 2012, at 10:28 AM, Josh Thompson wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> I think we targeted April to get a release out. The number of JIRA issues we
>> have left is starting to go down a fair amount. I'm almost finished with the
>> frontend. How's the backend? Are there other things we'd like to do for this
>> release such as scripting the install as much as possible? I don't think
>> we'll be done by the end of April, but I think the first week of May would be
>> possible.
>>
>> Aaron C., Jim, and David - Have you tested your provisioning modules with
>> recent code from subversion? If not, do you think you can do that in the next
>> week or two?
>>
>> Josh
>> - --
>> - -------------------------------
>> Josh Thompson
>> VCL Developer
>> North Carolina State University
>>
>> my GPG/PGP key can be found at pgp.mit.edu
>>
>> All electronic mail messages in connection with State business which
>> are sent to or received by this account are subject to the NC Public
>> Records Law and may be disclosed to third parties.
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v2.0.17 (GNU/Linux)
>>
>> iEYEARECAAYFAk+YChIACgkQV/LQcNdtPQPsagCdHCZOuT1ZIXxZrB+9EtppUjkc
>> KJIAn1mxEVtuZ64KEG0I4AwosUo3qT2P
>> =InjJ
>> -----END PGP SIGNATURE-----
>>
>
Re: status of 2.3 release
Posted by Aaron Coburn <ac...@amherst.edu>.
I looked through the current code for VCL::Module::Provisioning::VMware::vSphere_SDK
For the most part, all of the code I submitted for the vCenter module has been integrated into vSphere_SDK class.
There are three exceptions, which I didn't see in the current code:
1. in the move_virtual_disk subroutine, you use the MoveVirtualDisk vSphere method. This works fine on esx hosts, but in the context of a vCenter, it will return a "not implemented" fault. I liked how this was handled in the copy_virtual_disk subroutine: first try the method that makes sense and then fall back to using the full 'CloneVM' approach if the first method doesn't work. The way I had implemented this, move_virtual_disk just uses the copy_virtual_disk method and then deletes the source files upon successful completion. I can add that code if you would like.
2. in the copy_file_to and copy_file_from subroutines, I needed to add a loop because the VIExt::http_put_file and VIExt::http_get_file functions periodically fail. That is, if copy_file_to fails, the code sleeps for a few seconds and tries again, up to a maximum of a few tries. From what I can tell, this sort of looping doesn't exist when prepare_vmx() is called. I can wrap that in a Module::code_loop_timeout subroutine, if you would like.
3. When vCenter clones a VM with the CloneVM method, it will silently truncate long vm names. This isn't necessarily a problem as long as that new name finds its way into the database; otherwise, however, when VMX files are generated, the vmdk file pointers will reference incorrect locations. I had solved this by enforcing a maximum length in the web code (i.e. the vcl.image.name table). There are pros and cons to this, though a better approach may be to retrieve the actual name of the vmdk file after a clone operation is performed. This would require having the copy_virtual_disk and move_virtual_disk to return values other than just 1 (or, those methods could update the database themselves). I am not sure which approach you would prefer. I am also unsure of the best way to update the database with this information: would I simply call database_execute(...) and then call $self->data->refresh()? Or is there other code that handles this type of operation? Please advise.
Aaron
On Apr 25, 2012, at 10:28 AM, Josh Thompson wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I think we targeted April to get a release out. The number of JIRA issues we
> have left is starting to go down a fair amount. I'm almost finished with the
> frontend. How's the backend? Are there other things we'd like to do for this
> release such as scripting the install as much as possible? I don't think
> we'll be done by the end of April, but I think the first week of May would be
> possible.
>
> Aaron C., Jim, and David - Have you tested your provisioning modules with
> recent code from subversion? If not, do you think you can do that in the next
> week or two?
>
> Josh
> - --
> - -------------------------------
> Josh Thompson
> VCL Developer
> North Carolina State University
>
> my GPG/PGP key can be found at pgp.mit.edu
>
> All electronic mail messages in connection with State business which
> are sent to or received by this account are subject to the NC Public
> Records Law and may be disclosed to third parties.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (GNU/Linux)
>
> iEYEARECAAYFAk+YChIACgkQV/LQcNdtPQPsagCdHCZOuT1ZIXxZrB+9EtppUjkc
> KJIAn1mxEVtuZ64KEG0I4AwosUo3qT2P
> =InjJ
> -----END PGP SIGNATURE-----
>