[VMWARE]Any way to allow concurrent operations?

["Musayev, Ilya" <im...@webmd.net>]

If I need to do a mass operation on my vmware cluster, all jobs will be schedule and executed one at a time.

If you need to do a mass migration from one storage to another, this may take enormous amount of time. vSphere allows at least 4 concurrent operations and others are scheduled until the queue frees up.

Is there any possible method of allowing concurrent operations and not waiting for job X to complete before another one executes?

Thanks
ilya

RE: [VMWARE]Any way to allow concurrent operations?

["Musayev, Ilya" <im...@webmd.net>]

Kelven

Thanks for the response,

As it stands, the serial operations - even for storage maintenance - are unbearably slow - especially if your storage can handle many more concurrent tasks

Same applies for host maintenance mode, or any other operation like VM creation/deployment process. Everything is done in sequence. 

This approach is probably OK for small environments, but for really large implementations - when you need to spin instance up and down without waiting for a really long queue, or some other operation to complete .

For example, just now a colleague of mine added a windows template, at the same time I decide to provision more linux host. His windows template is about 5 GB in size, he kicked of his tasks for the first time, while I kicked off mine right after him.

As the result, I had to wait for his template to be initially deployed first, before my job can start. His job took about 20-30 minutes, my job takes less than 30 seconds. I understand this maybe a corner case, but our SLA can get really skewed if someone decides to add a new large sized template. If template additions are automated, we can go into a swing state on timing.

Like you mentioned, we need more concurrency.

Thanks
ilya

-----Original Message-----
From: Kelven Yang [mailto:kelven.yang@citrix.com] 
Sent: Tuesday, April 02, 2013 1:27 PM
To: dev@cloudstack.apache.org
Subject: Re: [VMWARE]Any way to allow concurrent operations?

We don't use queue facility/concurrency management within CloudStack extensively for now, the orchestration flow (i.e., putting storage into maintenance process) should launch concurrent tasks and manage it properly to achieve maximum of concurrency without breaking integrity of the operation. 

We do need to add more convenient facilities at framework level to enable these apparently needed features in various individual orchestration flows. Currently we write most of orchestration flows in a sequential execution manner.

Kelven

On 4/2/13 8:12 AM, "Marcus Sorensen" <sh...@gmail.com> wrote:

>Edison was saying something about executeInSequence, causing the serial 
>operations, and how it was a compatibility thing. He said he removed it 
>once as a test and it seemed to work for concurrent operations on KVM, 
>but I don't know much about it. Just thought it was worth mentioning.
>
>
>On Tue, Apr 2, 2013 at 9:06 AM, Chip Childers
><ch...@sungard.com>wrote:
>
>> On Tue, Apr 02, 2013 at 02:56:24PM +0000, Musayev, Ilya wrote:
>> > Long story short - due to the fact, we do things serially, any mass
>> operation will be queued, and may take very long time to complete.
>> >
>> > We need to improve this area, as its going to be one of the major
>> complaints in the near future for corporate users that run vmware.
>>
>> Sounds like something that we should create as an "Improvement" ticket.
>>

Re: [VMWARE]Any way to allow concurrent operations?

[Kelven Yang <ke...@citrix.com>]

We don't use queue facility/concurrency management within CloudStack
extensively for now, the orchestration flow (i.e., putting storage into
maintenance process) should launch concurrent tasks and manage it properly
to achieve maximum of concurrency without breaking integrity of the
operation. 

We do need to add more convenient facilities at framework level to enable
these apparently needed features in various individual orchestration
flows. Currently we write most of orchestration flows in a sequential
execution manner.

Kelven

On 4/2/13 8:12 AM, "Marcus Sorensen" <sh...@gmail.com> wrote:

>Edison was saying something about executeInSequence, causing the serial
>operations, and how it was a compatibility thing. He said he removed it
>once as a test and it seemed to work for concurrent operations on KVM, but
>I don't know much about it. Just thought it was worth mentioning.
>
>
>On Tue, Apr 2, 2013 at 9:06 AM, Chip Childers
><ch...@sungard.com>wrote:
>
>> On Tue, Apr 02, 2013 at 02:56:24PM +0000, Musayev, Ilya wrote:
>> > Long story short - due to the fact, we do things serially, any mass
>> operation will be queued, and may take very long time to complete.
>> >
>> > We need to improve this area, as its going to be one of the major
>> complaints in the near future for corporate users that run vmware.
>>
>> Sounds like something that we should create as an "Improvement" ticket.
>>

Re: [VMWARE]Any way to allow concurrent operations?

[Marcus Sorensen <sh...@gmail.com>]

Edison was saying something about executeInSequence, causing the serial
operations, and how it was a compatibility thing. He said he removed it
once as a test and it seemed to work for concurrent operations on KVM, but
I don't know much about it. Just thought it was worth mentioning.


On Tue, Apr 2, 2013 at 9:06 AM, Chip Childers <ch...@sungard.com>wrote:

> On Tue, Apr 02, 2013 at 02:56:24PM +0000, Musayev, Ilya wrote:
> > Long story short - due to the fact, we do things serially, any mass
> operation will be queued, and may take very long time to complete.
> >
> > We need to improve this area, as its going to be one of the major
> complaints in the near future for corporate users that run vmware.
>
> Sounds like something that we should create as an "Improvement" ticket.
>

Re: [VMWARE]Any way to allow concurrent operations?

[Chip Childers <ch...@sungard.com>]

On Tue, Apr 02, 2013 at 02:56:24PM +0000, Musayev, Ilya wrote:
> Long story short - due to the fact, we do things serially, any mass operation will be queued, and may take very long time to complete.
> 
> We need to improve this area, as its going to be one of the major complaints in the near future for corporate users that run vmware.

Sounds like something that we should create as an "Improvement" ticket.

RE: [VMWARE]Any way to allow concurrent operations?

["Musayev, Ilya" <im...@webmd.net>]

Long story short - due to the fact, we do things serially, any mass operation will be queued, and may take very long time to complete.

We need to improve this area, as its going to be one of the major complaints in the near future for corporate users that run vmware.

-----Original Message-----
From: Musayev, Ilya [mailto:imusayev@webmd.net] 
Sent: Tuesday, April 02, 2013 10:50 AM
To: dev@cloudstack.apache.org
Subject: RE: [VMWARE]Any way to allow concurrent operations?

Hi Chip,

Thanks for the feedback,

I'm well aware of that. But here is some food for thought.

I was running everything of NFS backend, primary and secondary stores.  As I was running out of disk space I decided to decommissioned my primary NFS store and use VMFS instead.

Here is a kicker, I've only had about 10 - 15 VMS in this env - very much lean with nothing but operating systems. This is POC, production will have several 100s.

My attempt to place NFS store in maintenance mode and bring up these VMs on VMFS - is still in progress. Its been over 10 hours. Mostly due to the fact that it runs 1 operation at a time.

Here is another scenario to try - more realistic, on the hypervisor that has many VMs, preferably 30 or more, place it into maintenance mode. This operation will take at least an hour or two depending on your setup - again due to the fact that we are running it in serial.

Even if we can bump up the number of threads to 2, this will greatly increase productivity. 

Regards
ilya

-----Original Message-----
From: Chip Childers [mailto:chip.childers@sungard.com] 
Sent: Tuesday, April 02, 2013 10:30 AM
To: dev@cloudstack.apache.org
Subject: Re: [VMWARE]Any way to allow concurrent operations?

On Tue, Apr 02, 2013 at 01:35:13AM +0000, Musayev, Ilya wrote:
> If I need to do a mass operation on my vmware cluster, all jobs will be schedule and executed one at a time.
> 
> If you need to do a mass migration from one storage to another, this may take enormous amount of time. vSphere allows at least 4 concurrent operations and others are scheduled until the queue frees up.
> 
> Is there any possible method of allowing concurrent operations and not waiting for job X to complete before another one executes?
> 
> Thanks
> ilya

AFAIK, CloudStack takes the approach of acting as the throttling mechanism itself for VMware interactions.  You're right that vSphere is able to handle some number of concurrent operations (depending on the operation, the target host and / or datastore, and the version of vSphere), and that it does queue up tasks in Virtual Center.  I can offer a warning though, that we should probably be careful about trying to make any change to the way CloudStack does this.  Virtual Center will actually time-out tasks that may not ever have a chance to run if they are pending for too long.  This may confuse CloudStack's logic in that scenario.

RE: [VMWARE]Any way to allow concurrent operations?

["Musayev, Ilya" <im...@webmd.net>]

Hi Chip,

Thanks for the feedback,

I'm well aware of that. But here is some food for thought.

I was running everything of NFS backend, primary and secondary stores.  As I was running out of disk space I decided to decommissioned my primary NFS store and use VMFS instead.

Here is a kicker, I've only had about 10 - 15 VMS in this env - very much lean with nothing but operating systems. This is POC, production will have several 100s.

My attempt to place NFS store in maintenance mode and bring up these VMs on VMFS - is still in progress. Its been over 10 hours. Mostly due to the fact that it runs 1 operation at a time.

Here is another scenario to try - more realistic, on the hypervisor that has many VMs, preferably 30 or more, place it into maintenance mode. This operation will take at least an hour or two depending on your setup - again due to the fact that we are running it in serial.

Even if we can bump up the number of threads to 2, this will greatly increase productivity. 

Regards
ilya

-----Original Message-----
From: Chip Childers [mailto:chip.childers@sungard.com] 
Sent: Tuesday, April 02, 2013 10:30 AM
To: dev@cloudstack.apache.org
Subject: Re: [VMWARE]Any way to allow concurrent operations?

On Tue, Apr 02, 2013 at 01:35:13AM +0000, Musayev, Ilya wrote:
> If I need to do a mass operation on my vmware cluster, all jobs will be schedule and executed one at a time.
> 
> If you need to do a mass migration from one storage to another, this may take enormous amount of time. vSphere allows at least 4 concurrent operations and others are scheduled until the queue frees up.
> 
> Is there any possible method of allowing concurrent operations and not waiting for job X to complete before another one executes?
> 
> Thanks
> ilya

AFAIK, CloudStack takes the approach of acting as the throttling mechanism itself for VMware interactions.  You're right that vSphere is able to handle some number of concurrent operations (depending on the operation, the target host and / or datastore, and the version of vSphere), and that it does queue up tasks in Virtual Center.  I can offer a warning though, that we should probably be careful about trying to make any change to the way CloudStack does this.  Virtual Center will actually time-out tasks that may not ever have a chance to run if they are pending for too long.  This may confuse CloudStack's logic in that scenario.

Re: [VMWARE]Any way to allow concurrent operations?

[Chip Childers <ch...@sungard.com>]

On Tue, Apr 02, 2013 at 01:35:13AM +0000, Musayev, Ilya wrote:
> If I need to do a mass operation on my vmware cluster, all jobs will be schedule and executed one at a time.
> 
> If you need to do a mass migration from one storage to another, this may take enormous amount of time. vSphere allows at least 4 concurrent operations and others are scheduled until the queue frees up.
> 
> Is there any possible method of allowing concurrent operations and not waiting for job X to complete before another one executes?
> 
> Thanks
> ilya

AFAIK, CloudStack takes the approach of acting as the throttling
mechanism itself for VMware interactions.  You're right that vSphere is
able to handle some number of concurrent operations (depending on the
operation, the target host and / or datastore, and the version of
vSphere), and that it does queue up tasks in Virtual Center.  I can
offer a warning though, that we should probably be careful about trying
to make any change to the way CloudStack does this.  Virtual Center will
actually time-out tasks that may not ever have a chance to run if they
are pending for too long.  This may confuse CloudStack's logic in that
scenario.