You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Rohit Yadav <ro...@shapeblue.com> on 2015/02/20 08:53:38 UTC
[DISCUSS] Improving VR services such as password server
Hi,
I'm trying to explore how to make systemvms more robust and
fault-tolerant, and the manual/automated QA of systemvms. One of the
common user facing issues related to scalability was the reset
password/key servers where the VR serves data using socat etc using
forking mechanisms and global locks. This slows down the processes such
as reset password.
More here: https://issues.apache.org/jira/browse/CLOUDSTACK-8272
One of the blindly thrown solutions includes increasing the VR RAM which
works for at scale but then seems to fail again when the load is
increased beyond a point. I don't know of any performance and stress
testing reports that tell us about these bottlenecks. Please share if
you have done anything in this regard.
I want to do couple of things:
- Explore systemvm build changes using newer tools such as packer
- Cleanup script execution and code in resource layer
- Start replacing bash scripts with more robust implementations, perhaps
a single or few agents on VRs that provide non-hardcoded well-documented
interfaces
- Right now everything in VR/systemvms is sort of hardcoded and the
services/interfaces are not well-documented. The idea is to refactor and
wrap everything we want to do with the systemvms in a general agents
framework that provides monitoring and managing the VRs (do stuff like
upgrades etc to combat things like ghost, poodle issues):
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Agents+Framework
What are the other issues you've had in past that you would like to be
improved?
--
Regards,
Rohit Yadav
Software Architect, ShapeBlue
M. +91 8826230892 | rohit.yadav@shapeblue.com
Blog: bhaisaab.org | Twitter: @_bhaisaab
PS. If you see any footer below, I did not add it :)
Find out more about ShapeBlue and our range of CloudStack related services
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
Re: [DISCUSS] Improving VR services such as password server
Posted by Funs Kessen <fu...@barred.org>.
Hi Rohit,
At the moment I’m looking into doing something with a much lighter and quite flexible alternative for the VR. I was actually thinking of using Tiny Core OS as a base, it has a great packaging system with loop mounts and is very simple and tiny compared to the VR we have now (tens of MBs instead of 1.x to 2GB). And upgrading it could easily be bundled from the systemvm.iso as it’s really small. Another thing I’m experimenting with is getting openvswitch in there which should enable us to get rid of some of the limitations with regard to the amounts of networks you can make (4 in a VPC due to the 7 nic limit ?) and some nice L2-ish features too, platform independent and perhaps move to full routing/switching functionality in the VR. It would also enable us to lift that functionality very simple into a container perhaps.
So the state right now is experimental, and I'm not at the point it could do all the singing and dancing, but I think it might be viable for further network virtualisation in CS and perhaps more.
Cheers,
Funs
> On 27 Mar 2015, at 07:37, Rohit Yadav <ro...@shapeblue.com> wrote:
>
> Hi everyone,
>
> I want to re-start the discussion on improving VR services and making a framework to build/improve old/new features such as reset password, dhcp, dns, vpn etc. that are both running on VR or on user VMs (reset password for example). For this, I’ve updated the wiki for high level goals and architecture diagram. Please share your comments and review on this proposal:
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Agents+Framework
>
> Kishan - Yes, I’m also want to have a mechanism to update/upgrade systemvms without say registering a new one, or say during the patching process it can upgrades packages; since recently we had seen several openssl related issues where a common fix was to simply upgrade the packages. See the above wiki for details.
>
>> On 20-Feb-2015, at 2:48 pm, Kishan Kavala <ki...@citrix.com> wrote:
>>
>> Rohit,
>> Completely agree with the scalability and maintenance issues with systems Vms. Using an agent inside VR will probably work well for KVM, since the KVM agent (Resource layer) is remote. Communication using link-local network also won't be a problem is such case.
>> For other hypervisors, it will increase load on the mgmt server due to direct agents. Also, mgmt server cannot access VRs directly.
>>
>> The options you mentioned are definitely worth exploring.
>>
>> Are you also looking at improving system Vm upgrade procedure?
>>
>> -----Original Message-----
>> From: Rohit Yadav [mailto:rohit.yadav@shapeblue.com]
>> Sent: Friday, February 20, 2015 1:24 PM
>> To: dev; users@cloudstack.apache.org
>> Subject: [DISCUSS] Improving VR services such as password server
>>
>> Hi,
>>
>> I'm trying to explore how to make systemvms more robust and fault-tolerant, and the manual/automated QA of systemvms. One of the common user facing issues related to scalability was the reset password/key servers where the VR serves data using socat etc using forking mechanisms and global locks. This slows down the processes such as reset password.
>>
>> More here: https://issues.apache.org/jira/browse/CLOUDSTACK-8272
>>
>> One of the blindly thrown solutions includes increasing the VR RAM which works for at scale but then seems to fail again when the load is increased beyond a point. I don't know of any performance and stress testing reports that tell us about these bottlenecks. Please share if you have done anything in this regard.
>>
>> I want to do couple of things:
>>
>> - Explore systemvm build changes using newer tools such as packer
>> - Cleanup script execution and code in resource layer
>> - Start replacing bash scripts with more robust implementations, perhaps a single or few agents on VRs that provide non-hardcoded well-documented interfaces
>> - Right now everything in VR/systemvms is sort of hardcoded and the services/interfaces are not well-documented. The idea is to refactor and wrap everything we want to do with the systemvms in a general agents framework that provides monitoring and managing the VRs (do stuff like upgrades etc to combat things like ghost, poodle issues):
>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Agents+Framework
>>
>> What are the other issues you've had in past that you would like to be improved?
>>
>> --
>> Regards,
>> Rohit Yadav
>> Software Architect, ShapeBlue
>> M. +91 8826230892 | rohit.yadav@shapeblue.com
>> Blog: bhaisaab.org | Twitter: @_bhaisaab PS. If you see any footer below, I did not add it :) Find out more about ShapeBlue and our range of CloudStack related services
>>
>> IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
>> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
>> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
>> CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
>> CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
>> CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
>>
>> This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
>
> Regards,
> Rohit Yadav
> Software Architect, ShapeBlue
> M. +91 88 262 30892 | rohit.yadav@shapeblue.com
> Blog: bhaisaab.org | Twitter: @_bhaisaab
>
>
>
> Find out more about ShapeBlue and our range of CloudStack related services
>
> IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
> CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
> CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
>
> This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
—
=Funs
Re: [DISCUSS] Improving VR services such as password server
Posted by ilya <il...@gmail.com>.
How about extending all features into io-socket in KVM/XEN and VM
Annotation details with VmWare?
Similar to what we do with system VMs when they come up - except there
are no router vms to manage and depend on.
On 3/26/15 11:37 PM, Rohit Yadav wrote:
> Hi everyone,
>
> I want to re-start the discussion on improving VR services and making a framework to build/improve old/new features such as reset password, dhcp, dns, vpn etc. that are both running on VR or on user VMs (reset password for example). For this, I’ve updated the wiki for high level goals and architecture diagram. Please share your comments and review on this proposal:
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Agents+Framework
>
> Kishan - Yes, I’m also want to have a mechanism to update/upgrade systemvms without say registering a new one, or say during the patching process it can upgrades packages; since recently we had seen several openssl related issues where a common fix was to simply upgrade the packages. See the above wiki for details.
>
>> On 20-Feb-2015, at 2:48 pm, Kishan Kavala <ki...@citrix.com> wrote:
>>
>> Rohit,
>> Completely agree with the scalability and maintenance issues with systems Vms. Using an agent inside VR will probably work well for KVM, since the KVM agent (Resource layer) is remote. Communication using link-local network also won't be a problem is such case.
>> For other hypervisors, it will increase load on the mgmt server due to direct agents. Also, mgmt server cannot access VRs directly.
>>
>> The options you mentioned are definitely worth exploring.
>>
>> Are you also looking at improving system Vm upgrade procedure?
>>
>> -----Original Message-----
>> From: Rohit Yadav [mailto:rohit.yadav@shapeblue.com]
>> Sent: Friday, February 20, 2015 1:24 PM
>> To: dev; users@cloudstack.apache.org
>> Subject: [DISCUSS] Improving VR services such as password server
>>
>> Hi,
>>
>> I'm trying to explore how to make systemvms more robust and fault-tolerant, and the manual/automated QA of systemvms. One of the common user facing issues related to scalability was the reset password/key servers where the VR serves data using socat etc using forking mechanisms and global locks. This slows down the processes such as reset password.
>>
>> More here: https://issues.apache.org/jira/browse/CLOUDSTACK-8272
>>
>> One of the blindly thrown solutions includes increasing the VR RAM which works for at scale but then seems to fail again when the load is increased beyond a point. I don't know of any performance and stress testing reports that tell us about these bottlenecks. Please share if you have done anything in this regard.
>>
>> I want to do couple of things:
>>
>> - Explore systemvm build changes using newer tools such as packer
>> - Cleanup script execution and code in resource layer
>> - Start replacing bash scripts with more robust implementations, perhaps a single or few agents on VRs that provide non-hardcoded well-documented interfaces
>> - Right now everything in VR/systemvms is sort of hardcoded and the services/interfaces are not well-documented. The idea is to refactor and wrap everything we want to do with the systemvms in a general agents framework that provides monitoring and managing the VRs (do stuff like upgrades etc to combat things like ghost, poodle issues):
>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Agents+Framework
>>
>> What are the other issues you've had in past that you would like to be improved?
>>
>> --
>> Regards,
>> Rohit Yadav
>> Software Architect, ShapeBlue
>> M. +91 8826230892 | rohit.yadav@shapeblue.com
>> Blog: bhaisaab.org | Twitter: @_bhaisaab PS. If you see any footer below, I did not add it :) Find out more about ShapeBlue and our range of CloudStack related services
>>
>> IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
>> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
>> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
>> CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
>> CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
>> CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
>>
>> This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
> Regards,
> Rohit Yadav
> Software Architect, ShapeBlue
> M. +91 88 262 30892 | rohit.yadav@shapeblue.com
> Blog: bhaisaab.org | Twitter: @_bhaisaab
>
>
>
> Find out more about ShapeBlue and our range of CloudStack related services
>
> IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
> CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
> CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
>
> This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
Re: [DISCUSS] Improving VR services such as password server
Posted by Rohit Yadav <ro...@shapeblue.com>.
Hi everyone,
I want to re-start the discussion on improving VR services and making a framework to build/improve old/new features such as reset password, dhcp, dns, vpn etc. that are both running on VR or on user VMs (reset password for example). For this, I’ve updated the wiki for high level goals and architecture diagram. Please share your comments and review on this proposal:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Agents+Framework
Kishan - Yes, I’m also want to have a mechanism to update/upgrade systemvms without say registering a new one, or say during the patching process it can upgrades packages; since recently we had seen several openssl related issues where a common fix was to simply upgrade the packages. See the above wiki for details.
> On 20-Feb-2015, at 2:48 pm, Kishan Kavala <ki...@citrix.com> wrote:
>
> Rohit,
> Completely agree with the scalability and maintenance issues with systems Vms. Using an agent inside VR will probably work well for KVM, since the KVM agent (Resource layer) is remote. Communication using link-local network also won't be a problem is such case.
> For other hypervisors, it will increase load on the mgmt server due to direct agents. Also, mgmt server cannot access VRs directly.
>
> The options you mentioned are definitely worth exploring.
>
> Are you also looking at improving system Vm upgrade procedure?
>
> -----Original Message-----
> From: Rohit Yadav [mailto:rohit.yadav@shapeblue.com]
> Sent: Friday, February 20, 2015 1:24 PM
> To: dev; users@cloudstack.apache.org
> Subject: [DISCUSS] Improving VR services such as password server
>
> Hi,
>
> I'm trying to explore how to make systemvms more robust and fault-tolerant, and the manual/automated QA of systemvms. One of the common user facing issues related to scalability was the reset password/key servers where the VR serves data using socat etc using forking mechanisms and global locks. This slows down the processes such as reset password.
>
> More here: https://issues.apache.org/jira/browse/CLOUDSTACK-8272
>
> One of the blindly thrown solutions includes increasing the VR RAM which works for at scale but then seems to fail again when the load is increased beyond a point. I don't know of any performance and stress testing reports that tell us about these bottlenecks. Please share if you have done anything in this regard.
>
> I want to do couple of things:
>
> - Explore systemvm build changes using newer tools such as packer
> - Cleanup script execution and code in resource layer
> - Start replacing bash scripts with more robust implementations, perhaps a single or few agents on VRs that provide non-hardcoded well-documented interfaces
> - Right now everything in VR/systemvms is sort of hardcoded and the services/interfaces are not well-documented. The idea is to refactor and wrap everything we want to do with the systemvms in a general agents framework that provides monitoring and managing the VRs (do stuff like upgrades etc to combat things like ghost, poodle issues):
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Agents+Framework
>
> What are the other issues you've had in past that you would like to be improved?
>
> --
> Regards,
> Rohit Yadav
> Software Architect, ShapeBlue
> M. +91 8826230892 | rohit.yadav@shapeblue.com
> Blog: bhaisaab.org | Twitter: @_bhaisaab PS. If you see any footer below, I did not add it :) Find out more about ShapeBlue and our range of CloudStack related services
>
> IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
> CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
> CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
>
> This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
Regards,
Rohit Yadav
Software Architect, ShapeBlue
M. +91 88 262 30892 | rohit.yadav@shapeblue.com
Blog: bhaisaab.org | Twitter: @_bhaisaab
Find out more about ShapeBlue and our range of CloudStack related services
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
RE: [DISCUSS] Improving VR services such as password server
Posted by Kishan Kavala <Ki...@citrix.com>.
Rohit,
Completely agree with the scalability and maintenance issues with systems Vms. Using an agent inside VR will probably work well for KVM, since the KVM agent (Resource layer) is remote. Communication using link-local network also won't be a problem is such case.
For other hypervisors, it will increase load on the mgmt server due to direct agents. Also, mgmt server cannot access VRs directly.
The options you mentioned are definitely worth exploring.
Are you also looking at improving system Vm upgrade procedure?
-----Original Message-----
From: Rohit Yadav [mailto:rohit.yadav@shapeblue.com]
Sent: Friday, February 20, 2015 1:24 PM
To: dev; users@cloudstack.apache.org
Subject: [DISCUSS] Improving VR services such as password server
Hi,
I'm trying to explore how to make systemvms more robust and fault-tolerant, and the manual/automated QA of systemvms. One of the common user facing issues related to scalability was the reset password/key servers where the VR serves data using socat etc using forking mechanisms and global locks. This slows down the processes such as reset password.
More here: https://issues.apache.org/jira/browse/CLOUDSTACK-8272
One of the blindly thrown solutions includes increasing the VR RAM which works for at scale but then seems to fail again when the load is increased beyond a point. I don't know of any performance and stress testing reports that tell us about these bottlenecks. Please share if you have done anything in this regard.
I want to do couple of things:
- Explore systemvm build changes using newer tools such as packer
- Cleanup script execution and code in resource layer
- Start replacing bash scripts with more robust implementations, perhaps a single or few agents on VRs that provide non-hardcoded well-documented interfaces
- Right now everything in VR/systemvms is sort of hardcoded and the services/interfaces are not well-documented. The idea is to refactor and wrap everything we want to do with the systemvms in a general agents framework that provides monitoring and managing the VRs (do stuff like upgrades etc to combat things like ghost, poodle issues):
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Agents+Framework
What are the other issues you've had in past that you would like to be improved?
--
Regards,
Rohit Yadav
Software Architect, ShapeBlue
M. +91 8826230892 | rohit.yadav@shapeblue.com
Blog: bhaisaab.org | Twitter: @_bhaisaab PS. If you see any footer below, I did not add it :) Find out more about ShapeBlue and our range of CloudStack related services
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
Re: [DISCUSS] Improving VR services such as password server
Posted by Nux! <nu...@li.nux.ro>.
Do you know of any plans/possibilities to completely get rid of the VR and move functionality - especially routing traffic - in host agents?
On another note, something more pluggable would be nice. One thing I'd really love to see is OpenVPN support (because anything else is just a world of pain).
Lucian
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
----- Original Message -----
> From: "Rohit Yadav" <ro...@shapeblue.com>
> To: "dev" <de...@cloudstack.apache.org>, users@cloudstack.apache.org
> Sent: Friday, 20 February, 2015 07:53:38
> Subject: [DISCUSS] Improving VR services such as password server
> Hi,
>
> I'm trying to explore how to make systemvms more robust and
> fault-tolerant, and the manual/automated QA of systemvms. One of the
> common user facing issues related to scalability was the reset
> password/key servers where the VR serves data using socat etc using
> forking mechanisms and global locks. This slows down the processes such
> as reset password.
>
> More here: https://issues.apache.org/jira/browse/CLOUDSTACK-8272
>
> One of the blindly thrown solutions includes increasing the VR RAM which
> works for at scale but then seems to fail again when the load is
> increased beyond a point. I don't know of any performance and stress
> testing reports that tell us about these bottlenecks. Please share if
> you have done anything in this regard.
>
> I want to do couple of things:
>
> - Explore systemvm build changes using newer tools such as packer
> - Cleanup script execution and code in resource layer
> - Start replacing bash scripts with more robust implementations, perhaps
> a single or few agents on VRs that provide non-hardcoded well-documented
> interfaces
> - Right now everything in VR/systemvms is sort of hardcoded and the
> services/interfaces are not well-documented. The idea is to refactor and
> wrap everything we want to do with the systemvms in a general agents
> framework that provides monitoring and managing the VRs (do stuff like
> upgrades etc to combat things like ghost, poodle issues):
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Agents+Framework
>
> What are the other issues you've had in past that you would like to be
> improved?
>
> --
> Regards,
> Rohit Yadav
> Software Architect, ShapeBlue
> M. +91 8826230892 | rohit.yadav@shapeblue.com
> Blog: bhaisaab.org | Twitter: @_bhaisaab
> PS. If you see any footer below, I did not add it :)
> Find out more about ShapeBlue and our range of CloudStack related services
>
> IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
> CloudStack Software
> Engineering<http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure
> Support<http://shapeblue.com/cloudstack-infrastructure-support/>
> CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
>
> This email and any attachments to it may be confidential and are intended solely
> for the use of the individual to whom it is addressed. Any views or opinions
> expressed are solely those of the author and do not necessarily represent those
> of Shape Blue Ltd or related companies. If you are not the intended recipient
> of this email, you must neither take any action based upon its contents, nor
> copy or show it to anyone. Please contact the sender if you believe you have
> received this email in error. Shape Blue Ltd is a company incorporated in
> England & Wales. ShapeBlue Services India LLP is a company incorporated in
> India and is operated under license from Shape Blue Ltd. Shape Blue Brasil
> Consultoria Ltda is a company incorporated in Brasil and is operated under
> license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by
> The Republic of South Africa and is traded under license from Shape Blue Ltd.
> ShapeBlue is a registered trademark.
Re: [DISCUSS] Improving VR services such as password server
Posted by Nux! <nu...@li.nux.ro>.
Do you know of any plans/possibilities to completely get rid of the VR and move functionality - especially routing traffic - in host agents?
On another note, something more pluggable would be nice. One thing I'd really love to see is OpenVPN support (because anything else is just a world of pain).
Lucian
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
----- Original Message -----
> From: "Rohit Yadav" <ro...@shapeblue.com>
> To: "dev" <de...@cloudstack.apache.org>, users@cloudstack.apache.org
> Sent: Friday, 20 February, 2015 07:53:38
> Subject: [DISCUSS] Improving VR services such as password server
> Hi,
>
> I'm trying to explore how to make systemvms more robust and
> fault-tolerant, and the manual/automated QA of systemvms. One of the
> common user facing issues related to scalability was the reset
> password/key servers where the VR serves data using socat etc using
> forking mechanisms and global locks. This slows down the processes such
> as reset password.
>
> More here: https://issues.apache.org/jira/browse/CLOUDSTACK-8272
>
> One of the blindly thrown solutions includes increasing the VR RAM which
> works for at scale but then seems to fail again when the load is
> increased beyond a point. I don't know of any performance and stress
> testing reports that tell us about these bottlenecks. Please share if
> you have done anything in this regard.
>
> I want to do couple of things:
>
> - Explore systemvm build changes using newer tools such as packer
> - Cleanup script execution and code in resource layer
> - Start replacing bash scripts with more robust implementations, perhaps
> a single or few agents on VRs that provide non-hardcoded well-documented
> interfaces
> - Right now everything in VR/systemvms is sort of hardcoded and the
> services/interfaces are not well-documented. The idea is to refactor and
> wrap everything we want to do with the systemvms in a general agents
> framework that provides monitoring and managing the VRs (do stuff like
> upgrades etc to combat things like ghost, poodle issues):
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Agents+Framework
>
> What are the other issues you've had in past that you would like to be
> improved?
>
> --
> Regards,
> Rohit Yadav
> Software Architect, ShapeBlue
> M. +91 8826230892 | rohit.yadav@shapeblue.com
> Blog: bhaisaab.org | Twitter: @_bhaisaab
> PS. If you see any footer below, I did not add it :)
> Find out more about ShapeBlue and our range of CloudStack related services
>
> IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
> CloudStack Software
> Engineering<http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure
> Support<http://shapeblue.com/cloudstack-infrastructure-support/>
> CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
>
> This email and any attachments to it may be confidential and are intended solely
> for the use of the individual to whom it is addressed. Any views or opinions
> expressed are solely those of the author and do not necessarily represent those
> of Shape Blue Ltd or related companies. If you are not the intended recipient
> of this email, you must neither take any action based upon its contents, nor
> copy or show it to anyone. Please contact the sender if you believe you have
> received this email in error. Shape Blue Ltd is a company incorporated in
> England & Wales. ShapeBlue Services India LLP is a company incorporated in
> India and is operated under license from Shape Blue Ltd. Shape Blue Brasil
> Consultoria Ltda is a company incorporated in Brasil and is operated under
> license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by
> The Republic of South Africa and is traded under license from Shape Blue Ltd.
> ShapeBlue is a registered trademark.
RE: [DISCUSS] Improving VR services such as password server
Posted by Kishan Kavala <Ki...@citrix.com>.
Rohit,
Completely agree with the scalability and maintenance issues with systems Vms. Using an agent inside VR will probably work well for KVM, since the KVM agent (Resource layer) is remote. Communication using link-local network also won't be a problem is such case.
For other hypervisors, it will increase load on the mgmt server due to direct agents. Also, mgmt server cannot access VRs directly.
The options you mentioned are definitely worth exploring.
Are you also looking at improving system Vm upgrade procedure?
-----Original Message-----
From: Rohit Yadav [mailto:rohit.yadav@shapeblue.com]
Sent: Friday, February 20, 2015 1:24 PM
To: dev; users@cloudstack.apache.org
Subject: [DISCUSS] Improving VR services such as password server
Hi,
I'm trying to explore how to make systemvms more robust and fault-tolerant, and the manual/automated QA of systemvms. One of the common user facing issues related to scalability was the reset password/key servers where the VR serves data using socat etc using forking mechanisms and global locks. This slows down the processes such as reset password.
More here: https://issues.apache.org/jira/browse/CLOUDSTACK-8272
One of the blindly thrown solutions includes increasing the VR RAM which works for at scale but then seems to fail again when the load is increased beyond a point. I don't know of any performance and stress testing reports that tell us about these bottlenecks. Please share if you have done anything in this regard.
I want to do couple of things:
- Explore systemvm build changes using newer tools such as packer
- Cleanup script execution and code in resource layer
- Start replacing bash scripts with more robust implementations, perhaps a single or few agents on VRs that provide non-hardcoded well-documented interfaces
- Right now everything in VR/systemvms is sort of hardcoded and the services/interfaces are not well-documented. The idea is to refactor and wrap everything we want to do with the systemvms in a general agents framework that provides monitoring and managing the VRs (do stuff like upgrades etc to combat things like ghost, poodle issues):
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Agents+Framework
What are the other issues you've had in past that you would like to be improved?
--
Regards,
Rohit Yadav
Software Architect, ShapeBlue
M. +91 8826230892 | rohit.yadav@shapeblue.com
Blog: bhaisaab.org | Twitter: @_bhaisaab PS. If you see any footer below, I did not add it :) Find out more about ShapeBlue and our range of CloudStack related services
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
Re: [DISCUSS] Improving VR services such as password server
Posted by Erik Weber <te...@gmail.com>.
On Fri, Feb 20, 2015 at 8:53 AM, Rohit Yadav <ro...@shapeblue.com>
wrote:
> Hi,
>
> I'm trying to explore how to make systemvms more robust and
> fault-tolerant, and the manual/automated QA of systemvms. One of the
> common user facing issues related to scalability was the reset
> password/key servers where the VR serves data using socat etc using
> forking mechanisms and global locks. This slows down the processes such
> as reset password.
>
> More here: https://issues.apache.org/jira/browse/CLOUDSTACK-8272
>
> One of the blindly thrown solutions includes increasing the VR RAM which
> works for at scale but then seems to fail again when the load is
> increased beyond a point. I don't know of any performance and stress
> testing reports that tell us about these bottlenecks. Please share if
> you have done anything in this regard.
>
>
Increasing the RAM is probably a perfectly fine solution if you're running
basic networking or have a small advanced infrastructure, but when you get
up to hundreds, or thousands of VRs even a bump of 128MB is considered
major.
(100 VRs + 128MB RAM = 12GB of additional RAM used).
Thus, keeping the footprint small is important.
> I want to do couple of things:
>
> - Explore systemvm build changes using newer tools such as packer
> - Cleanup script execution and code in resource layer
> - Start replacing bash scripts with more robust implementations, perhaps
> a single or few agents on VRs that provide non-hardcoded well-documented
> interfaces
> - Right now everything in VR/systemvms is sort of hardcoded and the
> services/interfaces are not well-documented. The idea is to refactor and
> wrap everything we want to do with the systemvms in a general agents
> framework that provides monitoring and managing the VRs (do stuff like
> upgrades etc to combat things like ghost, poodle issues):
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Agents+Framework
>
> What are the other issues you've had in past that you would like to be
> improved?
>
>
I like all those ideas. In general our pain areas are:
- extraordinary upgrades out of the ordinary version upgrades, poodle etc.
- upgrading during version upgrades. most of the times the changes in the
system vms are minor, and could be solved with some simple upgrade scripts
on the vms
--
Erik