You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@vcl.apache.org by momoh Ibrahim <mo...@gmail.com> on 2011/04/19 14:15:46 UTC
Blades running out of space
Hi Everyone,
My university have VCL (using vmware) running on 7 blades, it is difficult
to make reservations because the blades are out of space. I have a feeling
something is not being deleted after reservations are expired/completed. Any
suggestions on what to do?
Thank you.
Ibrahim Momoh
Re: Blades running out of space
Posted by IBRAHIM MOMOH <mo...@gmail.com>.
Thanks Alex.
On Apr 19, 2011, at 12:15 PM, Alexander Patterson wrote:
> Hello,
>
> We have this same issue at CSUEB. We have X amount of resources and X
> amount of limited local storage space on our current Blade hardware.
>
> For example, we have X core blades at X mhz per core with X gb of ram
> and X GB HD of local storage that is not upgradable on our current
> hardware. I have then calculated X number of VM's can run with X
> amount of performance to failure ratio.
>
> We are running 20 to 22 VM"s per blade that is about 1536mb of ram
> each and we give each VM tmp directory about 1 to 3 gb of space per
> VM. This makes each VM able to have 1 core at 2400mhz per VM, with our
> current hardware. We have our Vmware tmp space set to /dev/shm and
> have giving ourself 87 gb. We had it set to 40 GB but were also
> running out of space. We have the main images and VMguest load on to
> our netappfiler02:/vol/sata1/virtualmachines .
>
> Here is the our solution for changing the local VMware TMP directory
> on our blades
>
> -Add to /etc/fstab:
> none /dev/shm tmpfs size=87G 0 0 (You can set yours to whatever)
> -Run 'mount -a'
> -Add or edit the line in /etc/vmware/config:
> tmpDirectory="/dev/shm"
> -Restart VMware: 'service vmware restart'
> -Restart the VMs on the host
> -Verify that the memory-mapped files are being stored in /dev/shm:
> 'lsof | grep vmware-root'
> -You should see the space used on /dev/shm increase. Verify this by
> running 'df /dev/shm'.
>
> We are running a NFS share with our VCL, I also want to point that
> out. We have tried to use the nFS share as the tmpDirectory for
> VMware, this lead to failures and very slow speeds, the Vmware
> tmpDirectory should be hosted local if you have the space.
>
> I have noticed in testing as more software or data is loaded the VM's
> increases in size on both the copy image (Vmguest-XX) this is storage
> on our local NFS mount and the local blade VM directory. In campus
> testing we have found out the hard way with this method due to
> failures poor performance when reloading mutipal VM's on our small
> amount of blades. We had originaly set our system to run 30 VM"s per
> blades but ran out of resources and local storage space. When the
> system reloads 30 to 50 + VM's it's going to pushing X amount of
> network space/local storage ect. If you run out of tmp space on your
> local blade you get all failures on all reservations once your hit the
> limit. Also when you reload 10 or 15 VM's at one time on a blade, It
> can get very slow. We don't have 100's of blades like NC state, to
> prevision and ease the load on our hardware, we had to cut the number
> of VM's per blades to increase performance. To help with consistence
> reloading, we have also changed some of the Virtual Environments with
> 4 or 5 softwares into one image; so multiple classes can use the same
> image and not have to reload a image for each class. This has helped
> tremendously here on our campus.
>
> Here is an example of our local storage setup on one of our Blades
> here. We are also using VCL 2.1 and Vmware 1.X with our older setup.
>
> [root@vclesx04 ~]# df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda2 7.8G 3.6G 3.8G 49% /
> /dev/sda6 3.8G 571M 3.1G 16% /var
> /dev/sda5 3.8G 188M 3.5G 6% /home
> /dev/sda1 97M 18M 75M 19% /boot
> tmpfs 87G 36G 52G 42% /dev/shm
> netappfiler02:/vol/sata1/virtualmachines
> 5.3T 2.2T 3.1T 42% /virtualmachines
>
> -Alex
>
>
> On Tue, Apr 19, 2011 at 9:26 AM, IBRAHIM MOMOH <mo...@gmail.com> wrote:
>> hi Andy,
>>
>> Thanks for your quick response. Here are the requested details:
>>
>> vm guest files are taking up all the space. i think they should be deleted after a session ends.
>> Provisioning module: Vmware server
>> VCL Version: VCL2.1
>> VMware version: 1.0.10
>>
>>
>> On Apr 19, 2011, at 10:05 AM, Andy Kurth wrote:
>>
>>> Hello,
>>> More information is necessary. Which files are taking up the space? Which provisioning module are you using? Which version of VCL and VMware are you running?
>>>
>>> -Andy
>>>
>>> On 4/19/2011 8:15 AM, momoh Ibrahim wrote:
>>>> Hi Everyone,
>>>>
>>>> My university have VCL (using vmware) running on 7 blades, it is
>>>> difficult to make reservations because the blades are out of space. I
>>>> have a feeling something is not being deleted after reservations are
>>>> expired/completed. Any suggestions on what to do?
>>>>
>>>> Thank you.
>>>>
>>>>
>>>> Ibrahim Momoh
>>>>
>>
>>
>
>
>
> --
> Thanks,
> Alex Patterson
> User Support Services
> Operating System Analyst
> California State University, East Bay
Re: Blades running out of space
Posted by Alexander Patterson <al...@csueastbay.edu>.
Hello,
We have this same issue at CSUEB. We have X amount of resources and X
amount of limited local storage space on our current Blade hardware.
For example, we have X core blades at X mhz per core with X gb of ram
and X GB HD of local storage that is not upgradable on our current
hardware. I have then calculated X number of VM's can run with X
amount of performance to failure ratio.
We are running 20 to 22 VM"s per blade that is about 1536mb of ram
each and we give each VM tmp directory about 1 to 3 gb of space per
VM. This makes each VM able to have 1 core at 2400mhz per VM, with our
current hardware. We have our Vmware tmp space set to /dev/shm and
have giving ourself 87 gb. We had it set to 40 GB but were also
running out of space. We have the main images and VMguest load on to
our netappfiler02:/vol/sata1/virtualmachines .
Here is the our solution for changing the local VMware TMP directory
on our blades
-Add to /etc/fstab:
none /dev/shm tmpfs size=87G 0 0 (You can set yours to whatever)
-Run 'mount -a'
-Add or edit the line in /etc/vmware/config:
tmpDirectory="/dev/shm"
-Restart VMware: 'service vmware restart'
-Restart the VMs on the host
-Verify that the memory-mapped files are being stored in /dev/shm:
'lsof | grep vmware-root'
-You should see the space used on /dev/shm increase. Verify this by
running 'df /dev/shm'.
We are running a NFS share with our VCL, I also want to point that
out. We have tried to use the nFS share as the tmpDirectory for
VMware, this lead to failures and very slow speeds, the Vmware
tmpDirectory should be hosted local if you have the space.
I have noticed in testing as more software or data is loaded the VM's
increases in size on both the copy image (Vmguest-XX) this is storage
on our local NFS mount and the local blade VM directory. In campus
testing we have found out the hard way with this method due to
failures poor performance when reloading mutipal VM's on our small
amount of blades. We had originaly set our system to run 30 VM"s per
blades but ran out of resources and local storage space. When the
system reloads 30 to 50 + VM's it's going to pushing X amount of
network space/local storage ect. If you run out of tmp space on your
local blade you get all failures on all reservations once your hit the
limit. Also when you reload 10 or 15 VM's at one time on a blade, It
can get very slow. We don't have 100's of blades like NC state, to
prevision and ease the load on our hardware, we had to cut the number
of VM's per blades to increase performance. To help with consistence
reloading, we have also changed some of the Virtual Environments with
4 or 5 softwares into one image; so multiple classes can use the same
image and not have to reload a image for each class. This has helped
tremendously here on our campus.
Here is an example of our local storage setup on one of our Blades
here. We are also using VCL 2.1 and Vmware 1.X with our older setup.
[root@vclesx04 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 7.8G 3.6G 3.8G 49% /
/dev/sda6 3.8G 571M 3.1G 16% /var
/dev/sda5 3.8G 188M 3.5G 6% /home
/dev/sda1 97M 18M 75M 19% /boot
tmpfs 87G 36G 52G 42% /dev/shm
netappfiler02:/vol/sata1/virtualmachines
5.3T 2.2T 3.1T 42% /virtualmachines
-Alex
On Tue, Apr 19, 2011 at 9:26 AM, IBRAHIM MOMOH <mo...@gmail.com> wrote:
> hi Andy,
>
> Thanks for your quick response. Here are the requested details:
>
> vm guest files are taking up all the space. i think they should be deleted after a session ends.
> Provisioning module: Vmware server
> VCL Version: VCL2.1
> VMware version: 1.0.10
>
>
> On Apr 19, 2011, at 10:05 AM, Andy Kurth wrote:
>
>> Hello,
>> More information is necessary. Which files are taking up the space? Which provisioning module are you using? Which version of VCL and VMware are you running?
>>
>> -Andy
>>
>> On 4/19/2011 8:15 AM, momoh Ibrahim wrote:
>>> Hi Everyone,
>>>
>>> My university have VCL (using vmware) running on 7 blades, it is
>>> difficult to make reservations because the blades are out of space. I
>>> have a feeling something is not being deleted after reservations are
>>> expired/completed. Any suggestions on what to do?
>>>
>>> Thank you.
>>>
>>>
>>> Ibrahim Momoh
>>>
>
>
--
Thanks,
Alex Patterson
User Support Services
Operating System Analyst
California State University, East Bay
Re: Blades running out of space
Posted by IBRAHIM MOMOH <mo...@gmail.com>.
hi Andy,
Thanks for your quick response. Here are the requested details:
vm guest files are taking up all the space. i think they should be deleted after a session ends.
Provisioning module: Vmware server
VCL Version: VCL2.1
VMware version: 1.0.10
On Apr 19, 2011, at 10:05 AM, Andy Kurth wrote:
> Hello,
> More information is necessary. Which files are taking up the space? Which provisioning module are you using? Which version of VCL and VMware are you running?
>
> -Andy
>
> On 4/19/2011 8:15 AM, momoh Ibrahim wrote:
>> Hi Everyone,
>>
>> My university have VCL (using vmware) running on 7 blades, it is
>> difficult to make reservations because the blades are out of space. I
>> have a feeling something is not being deleted after reservations are
>> expired/completed. Any suggestions on what to do?
>>
>> Thank you.
>>
>>
>> Ibrahim Momoh
>>
Re: Blades running out of space
Posted by Andy Kurth <an...@ncsu.edu>.
Hello,
More information is necessary. Which files are taking up the space?
Which provisioning module are you using? Which version of VCL and
VMware are you running?
-Andy
On 4/19/2011 8:15 AM, momoh Ibrahim wrote:
> Hi Everyone,
>
> My university have VCL (using vmware) running on 7 blades, it is
> difficult to make reservations because the blades are out of space. I
> have a feeling something is not being deleted after reservations are
> expired/completed. Any suggestions on what to do?
>
> Thank you.
>
>
> Ibrahim Momoh
>