You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Indra Pramana <in...@sg.or.id> on 2013/07/20 09:18:12 UTC
CloudStack performance issue
Dear all,
On my test-lab environment, I encountered some cases of CloudStack
performance issue where creation or deleting VMs will become very slow.
This normally happens when I tried to do two or three simultaneous tasks at
the same time. I would need to wait 10-15 minutes just for a VM to be
created, started, stopped or destroyed.
Sometimes, certain tasks will not be completed at all and I need to reboot
the CloudStack management server for the tasks to complete. Some tasks
might fail, but will be OK when I try to attempt again. Sometimes, it will
cause the KVM hypervisor host to be in "discrd onnected" state and I have
to reboot the host.
Where can I start troubleshooting, which logs I need to go through to find
out the bottleneck? The management-server.log file doesn't show much
anything about the delay. It seems that there is some "locking" mechanism
and certain processes will need other processes to complete before they can
start. This is worrying since in a production environment, many customers
might want to try creating VMs at the same time simultaneously.
Any thoughts or feedback on this matter is appreciated.
Looking forward to your reply, thank you.
Cheers.
Re: CloudStack performance issue
Posted by Indra Pramana <in...@sg.or.id>.
Hi David,
Good day to you, and thank you for your e-mail.
Most of my templates are on 100 GB root disk, which I understand is too
big. I plan to recreate the templates on a 10 GB disk, which should be
sufficient based on an advice from another member of this mailing list on
another thread.
Do you think that might be the cause? I hope that deployment time will be
much faster after I recreated the templates.
Thank you.
On Wed, Jul 24, 2013 at 1:22 AM, David Ortiz <dp...@outlook.com> wrote:
> Indra,
> How big are the templates you are attempting to deploy? I could see
> the 1 Gbps connection being a little bit of a bottleneck, but it sounds
> like there may be something else going on. I know we would have issues on
> our setup which is on 1 Gbps connections at the moment if we tried to do
> multiple snapshots at once, but we could normally get away with deploying a
> couple VMs at once within a few minutes. As for the question about how
> they are being copied, I am not 100% confident on this, but I believe it is
> copied via the SSVM, so it would be going secondary storage --> hypervisor
> host --> primary storage. Hopefully someone will correct me if I am wrong
> on that.
> -David
>
> > Date: Tue, 23 Jul 2013 12:33:17 +0800
> > Subject: Re: CloudStack performance issue
> > From: indra@sg.or.id
> > To: users@cloudstack.apache.org
> >
> > Hi David,
> >
> > Good day to you, and thank you for your e-mail.
> >
> > My secondary storage is connected to the switch using 1 Gbps connection,
> > while my primary storage is connected via a 10 Gbps connection.
> > Unfortunately I don't have 10G NIC cards on my secondary storage. Do you
> > think upgrading my secondary storage's NIC to 10 Gbps will help?
> >
> > May I know how the template and ISOs are being copied, is it through the
> > hypervisor host (e.g. secondary storage --> hypervisor host --> primary
> > storage) or directly (i.e. secondary storage --> primary storage)? My
> > primary storage is not on the same subnet/VLAN as secondary storage,
> since
> > it's connected to different NIC cards on the hypervisor hosts. If it's a
> > direct connection between secondary storage to primary storage without
> > going to the hypervisor, I might be considering changing my network
> > topology to make the primary and secondary storage to be on the same
> > subnet/VLAN.
> >
> > Looking forward to your reply, thank you.
> >
> > Cheers.
> >
> >
> > On Tue, Jul 23, 2013 at 3:19 AM, David Ortiz <dp...@outlook.com>
> wrote:
> >
> > > Meant that question for Indra. Sorry.
> > >
> > > > From: dportiz@outlook.com
> > > > To: users@cloudstack.apache.org
> > > > Subject: RE: CloudStack performance issue
> > > > Date: Mon, 22 Jul 2013 11:34:10 -0400
> > > >
> > > > Prasanna,
> > > > What kind of network do you use for your setup? Creating a new
> VM
> > > requires copying the template or ISO over from secondary storage to
> > > primary, so that could be a bottleneck when trying to create multiple
> VMs
> > > at once.
> > > > -Dave
> > > >
> > > > > Date: Sat, 20 Jul 2013 16:20:08 +0530
> > > > > From: tsp@apache.org
> > > > > To: users@cloudstack.apache.org
> > > > > Subject: Re: CloudStack performance issue
> > > > >
> > > > > On Sat, Jul 20, 2013 at 03:18:12PM +0800, Indra Pramana wrote:
> > > > > >
> > > > > > Where can I start troubleshooting, which logs I need to go
> through
> > > to find
> > > > > > out the bottleneck? The management-server.log file doesn't show
> much
> > > > > > anything about the delay. It seems that there is some "locking"
> > > mechanism
> > > > > > and certain processes will need other processes to complete
> before
> > > they can
> > > > > > start. This is worrying since in a production environment, many
> > > customers
> > > > > > might want to try creating VMs at the same time simultaneously.
> > > > > >
> > > > >
> > > > > The logs of use would be managment-server.log which defaults to
> DEBUG
> > > > > mode. The logs on the agent (inside KVM host) are default in INFO
> > > > > mode. You can turn them up to DEBUG by altering the log4j config.
> > > > > Apart from that you can also look at libvirt logs. Folks have been
> > > > > tuning the deployment of the VMs on KVM host in another thread on
> dev@
> > > > > and in the bug
> https://issues.apache.org/jira/browse/CLOUDSTACK-3163
> > > > >
> > > > > let us know if the symptoms are the same and you can bring this up
> on
> > > > > dev@ with more details.
> > > > >
> > > > > > Any thoughts or feedback on this matter is appreciated.
> > > > > >
> > > > > > Looking forward to your reply, thank you.
> > > > > >
> > > > > > Cheers.
> > > > >
> > > > > --
> > > > > Prasanna.,
> > > > >
> > > > > ------------------------
> > > > > Powered by BigRock.com
> > > > >
> > > >
> > >
> > >
>
>
RE: CloudStack performance issue
Posted by David Ortiz <dp...@outlook.com>.
Indra,
How big are the templates you are attempting to deploy? I could see the 1 Gbps connection being a little bit of a bottleneck, but it sounds like there may be something else going on. I know we would have issues on our setup which is on 1 Gbps connections at the moment if we tried to do multiple snapshots at once, but we could normally get away with deploying a couple VMs at once within a few minutes. As for the question about how they are being copied, I am not 100% confident on this, but I believe it is copied via the SSVM, so it would be going secondary storage --> hypervisor host --> primary storage. Hopefully someone will correct me if I am wrong on that.
-David
> Date: Tue, 23 Jul 2013 12:33:17 +0800
> Subject: Re: CloudStack performance issue
> From: indra@sg.or.id
> To: users@cloudstack.apache.org
>
> Hi David,
>
> Good day to you, and thank you for your e-mail.
>
> My secondary storage is connected to the switch using 1 Gbps connection,
> while my primary storage is connected via a 10 Gbps connection.
> Unfortunately I don't have 10G NIC cards on my secondary storage. Do you
> think upgrading my secondary storage's NIC to 10 Gbps will help?
>
> May I know how the template and ISOs are being copied, is it through the
> hypervisor host (e.g. secondary storage --> hypervisor host --> primary
> storage) or directly (i.e. secondary storage --> primary storage)? My
> primary storage is not on the same subnet/VLAN as secondary storage, since
> it's connected to different NIC cards on the hypervisor hosts. If it's a
> direct connection between secondary storage to primary storage without
> going to the hypervisor, I might be considering changing my network
> topology to make the primary and secondary storage to be on the same
> subnet/VLAN.
>
> Looking forward to your reply, thank you.
>
> Cheers.
>
>
> On Tue, Jul 23, 2013 at 3:19 AM, David Ortiz <dp...@outlook.com> wrote:
>
> > Meant that question for Indra. Sorry.
> >
> > > From: dportiz@outlook.com
> > > To: users@cloudstack.apache.org
> > > Subject: RE: CloudStack performance issue
> > > Date: Mon, 22 Jul 2013 11:34:10 -0400
> > >
> > > Prasanna,
> > > What kind of network do you use for your setup? Creating a new VM
> > requires copying the template or ISO over from secondary storage to
> > primary, so that could be a bottleneck when trying to create multiple VMs
> > at once.
> > > -Dave
> > >
> > > > Date: Sat, 20 Jul 2013 16:20:08 +0530
> > > > From: tsp@apache.org
> > > > To: users@cloudstack.apache.org
> > > > Subject: Re: CloudStack performance issue
> > > >
> > > > On Sat, Jul 20, 2013 at 03:18:12PM +0800, Indra Pramana wrote:
> > > > >
> > > > > Where can I start troubleshooting, which logs I need to go through
> > to find
> > > > > out the bottleneck? The management-server.log file doesn't show much
> > > > > anything about the delay. It seems that there is some "locking"
> > mechanism
> > > > > and certain processes will need other processes to complete before
> > they can
> > > > > start. This is worrying since in a production environment, many
> > customers
> > > > > might want to try creating VMs at the same time simultaneously.
> > > > >
> > > >
> > > > The logs of use would be managment-server.log which defaults to DEBUG
> > > > mode. The logs on the agent (inside KVM host) are default in INFO
> > > > mode. You can turn them up to DEBUG by altering the log4j config.
> > > > Apart from that you can also look at libvirt logs. Folks have been
> > > > tuning the deployment of the VMs on KVM host in another thread on dev@
> > > > and in the bug https://issues.apache.org/jira/browse/CLOUDSTACK-3163
> > > >
> > > > let us know if the symptoms are the same and you can bring this up on
> > > > dev@ with more details.
> > > >
> > > > > Any thoughts or feedback on this matter is appreciated.
> > > > >
> > > > > Looking forward to your reply, thank you.
> > > > >
> > > > > Cheers.
> > > >
> > > > --
> > > > Prasanna.,
> > > >
> > > > ------------------------
> > > > Powered by BigRock.com
> > > >
> > >
> >
> >
Re: CloudStack performance issue
Posted by Indra Pramana <in...@sg.or.id>.
Hi David,
Good day to you, and thank you for your e-mail.
My secondary storage is connected to the switch using 1 Gbps connection,
while my primary storage is connected via a 10 Gbps connection.
Unfortunately I don't have 10G NIC cards on my secondary storage. Do you
think upgrading my secondary storage's NIC to 10 Gbps will help?
May I know how the template and ISOs are being copied, is it through the
hypervisor host (e.g. secondary storage --> hypervisor host --> primary
storage) or directly (i.e. secondary storage --> primary storage)? My
primary storage is not on the same subnet/VLAN as secondary storage, since
it's connected to different NIC cards on the hypervisor hosts. If it's a
direct connection between secondary storage to primary storage without
going to the hypervisor, I might be considering changing my network
topology to make the primary and secondary storage to be on the same
subnet/VLAN.
Looking forward to your reply, thank you.
Cheers.
On Tue, Jul 23, 2013 at 3:19 AM, David Ortiz <dp...@outlook.com> wrote:
> Meant that question for Indra. Sorry.
>
> > From: dportiz@outlook.com
> > To: users@cloudstack.apache.org
> > Subject: RE: CloudStack performance issue
> > Date: Mon, 22 Jul 2013 11:34:10 -0400
> >
> > Prasanna,
> > What kind of network do you use for your setup? Creating a new VM
> requires copying the template or ISO over from secondary storage to
> primary, so that could be a bottleneck when trying to create multiple VMs
> at once.
> > -Dave
> >
> > > Date: Sat, 20 Jul 2013 16:20:08 +0530
> > > From: tsp@apache.org
> > > To: users@cloudstack.apache.org
> > > Subject: Re: CloudStack performance issue
> > >
> > > On Sat, Jul 20, 2013 at 03:18:12PM +0800, Indra Pramana wrote:
> > > >
> > > > Where can I start troubleshooting, which logs I need to go through
> to find
> > > > out the bottleneck? The management-server.log file doesn't show much
> > > > anything about the delay. It seems that there is some "locking"
> mechanism
> > > > and certain processes will need other processes to complete before
> they can
> > > > start. This is worrying since in a production environment, many
> customers
> > > > might want to try creating VMs at the same time simultaneously.
> > > >
> > >
> > > The logs of use would be managment-server.log which defaults to DEBUG
> > > mode. The logs on the agent (inside KVM host) are default in INFO
> > > mode. You can turn them up to DEBUG by altering the log4j config.
> > > Apart from that you can also look at libvirt logs. Folks have been
> > > tuning the deployment of the VMs on KVM host in another thread on dev@
> > > and in the bug https://issues.apache.org/jira/browse/CLOUDSTACK-3163
> > >
> > > let us know if the symptoms are the same and you can bring this up on
> > > dev@ with more details.
> > >
> > > > Any thoughts or feedback on this matter is appreciated.
> > > >
> > > > Looking forward to your reply, thank you.
> > > >
> > > > Cheers.
> > >
> > > --
> > > Prasanna.,
> > >
> > > ------------------------
> > > Powered by BigRock.com
> > >
> >
>
>
RE: CloudStack performance issue
Posted by David Ortiz <dp...@outlook.com>.
Meant that question for Indra. Sorry.
> From: dportiz@outlook.com
> To: users@cloudstack.apache.org
> Subject: RE: CloudStack performance issue
> Date: Mon, 22 Jul 2013 11:34:10 -0400
>
> Prasanna,
> What kind of network do you use for your setup? Creating a new VM requires copying the template or ISO over from secondary storage to primary, so that could be a bottleneck when trying to create multiple VMs at once.
> -Dave
>
> > Date: Sat, 20 Jul 2013 16:20:08 +0530
> > From: tsp@apache.org
> > To: users@cloudstack.apache.org
> > Subject: Re: CloudStack performance issue
> >
> > On Sat, Jul 20, 2013 at 03:18:12PM +0800, Indra Pramana wrote:
> > >
> > > Where can I start troubleshooting, which logs I need to go through to find
> > > out the bottleneck? The management-server.log file doesn't show much
> > > anything about the delay. It seems that there is some "locking" mechanism
> > > and certain processes will need other processes to complete before they can
> > > start. This is worrying since in a production environment, many customers
> > > might want to try creating VMs at the same time simultaneously.
> > >
> >
> > The logs of use would be managment-server.log which defaults to DEBUG
> > mode. The logs on the agent (inside KVM host) are default in INFO
> > mode. You can turn them up to DEBUG by altering the log4j config.
> > Apart from that you can also look at libvirt logs. Folks have been
> > tuning the deployment of the VMs on KVM host in another thread on dev@
> > and in the bug https://issues.apache.org/jira/browse/CLOUDSTACK-3163
> >
> > let us know if the symptoms are the same and you can bring this up on
> > dev@ with more details.
> >
> > > Any thoughts or feedback on this matter is appreciated.
> > >
> > > Looking forward to your reply, thank you.
> > >
> > > Cheers.
> >
> > --
> > Prasanna.,
> >
> > ------------------------
> > Powered by BigRock.com
> >
>
RE: CloudStack performance issue
Posted by David Ortiz <dp...@outlook.com>.
Prasanna,
What kind of network do you use for your setup? Creating a new VM requires copying the template or ISO over from secondary storage to primary, so that could be a bottleneck when trying to create multiple VMs at once.
-Dave
> Date: Sat, 20 Jul 2013 16:20:08 +0530
> From: tsp@apache.org
> To: users@cloudstack.apache.org
> Subject: Re: CloudStack performance issue
>
> On Sat, Jul 20, 2013 at 03:18:12PM +0800, Indra Pramana wrote:
> >
> > Where can I start troubleshooting, which logs I need to go through to find
> > out the bottleneck? The management-server.log file doesn't show much
> > anything about the delay. It seems that there is some "locking" mechanism
> > and certain processes will need other processes to complete before they can
> > start. This is worrying since in a production environment, many customers
> > might want to try creating VMs at the same time simultaneously.
> >
>
> The logs of use would be managment-server.log which defaults to DEBUG
> mode. The logs on the agent (inside KVM host) are default in INFO
> mode. You can turn them up to DEBUG by altering the log4j config.
> Apart from that you can also look at libvirt logs. Folks have been
> tuning the deployment of the VMs on KVM host in another thread on dev@
> and in the bug https://issues.apache.org/jira/browse/CLOUDSTACK-3163
>
> let us know if the symptoms are the same and you can bring this up on
> dev@ with more details.
>
> > Any thoughts or feedback on this matter is appreciated.
> >
> > Looking forward to your reply, thank you.
> >
> > Cheers.
>
> --
> Prasanna.,
>
> ------------------------
> Powered by BigRock.com
>
Re: CloudStack performance issue
Posted by Prasanna Santhanam <ts...@apache.org>.
On Sat, Jul 20, 2013 at 03:18:12PM +0800, Indra Pramana wrote:
>
> Where can I start troubleshooting, which logs I need to go through to find
> out the bottleneck? The management-server.log file doesn't show much
> anything about the delay. It seems that there is some "locking" mechanism
> and certain processes will need other processes to complete before they can
> start. This is worrying since in a production environment, many customers
> might want to try creating VMs at the same time simultaneously.
>
The logs of use would be managment-server.log which defaults to DEBUG
mode. The logs on the agent (inside KVM host) are default in INFO
mode. You can turn them up to DEBUG by altering the log4j config.
Apart from that you can also look at libvirt logs. Folks have been
tuning the deployment of the VMs on KVM host in another thread on dev@
and in the bug https://issues.apache.org/jira/browse/CLOUDSTACK-3163
let us know if the symptoms are the same and you can bring this up on
dev@ with more details.
> Any thoughts or feedback on this matter is appreciated.
>
> Looking forward to your reply, thank you.
>
> Cheers.
--
Prasanna.,
------------------------
Powered by BigRock.com