You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Ivan Kudryavtsev <ku...@bw-sw.com> on 2018/10/22 18:02:58 UTC
KVM CloudStack Agent Hacking proposal
Hello, Devs.
I would like to introduce a feature and decided to consult with you about
its design before implementation. The feature is connected with KVM
CloudStack agent. We have found it beneficial to be able to launch custom
scripts upon VM start/stop. It can be done using Qemu hook but it has
several drawbacks:
- the hook is deployed by CS and adding additional lines into it leads to
extra efforts when ACS package is updated.
- it leads to deadlocks as you cannot effectively and easy to communicate
with libvirt from hook even with "fork & exec" because security_groups.py
and agent also participate and as a result it causes deadlocks.
Now, in the code, we have a call for "security_groups.py":
Start:
https://github.com/apache/cloudstack/blob/65f31f1a9fbc1c20cd752d80a7e1117efc0248a5/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtStartCommandWrapper.java#L103
Stop:
https://github.com/apache/cloudstack/blob/65f31f1a9fbc1c20cd752d80a7e1117efc0248a5/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtStopCommandWrapper.java#L88
I would like is to introduce a more generic approach, so the administrator
can specify additional scripts in the agent.properties, which will be
called the same way "security_groups.py" called.
custom.vm.start=/path/to/script1,path/to.script2
custom.vm.stop=/path/to/script3,path/to.script4
So, this feature will help users to do custom hotplug mechanisms. E.g. we
have such implementation which adds per-account VXLAN as a hotplug ethernet
device. So, even for a Basic Zone, every VM gets automatic second NIC which
helps to build a private network for an account.
Currently, we do the job thru adding lines into security_groups.py, which
is not a good approach, especially for end users who don't want to hack the
system.
Also, I'm thinking about changing /etc/libvirt/hooks/qemu the same way, so
it was just an entry point to /etc/libvirt/hooks/qemu.d/* located scripts.
Let me know about this feature proposal and if its design is good, we start
developing it.
Have a good day.
--
With best regards, Ivan Kudryavtsev
Bitworks LLC
Cell RU: +7-923-414-1515
Cell USA: +1-201-257-1512
WWW: http://bitworks.software/ <http://bw-sw.com/>
RE: KVM CloudStack Agent Hacking proposal
Posted by Paul Angus <pa...@shapeblue.com>.
Cool. Thanks for the explanation Ivan, I'll watch with interest.
paul.angus@shapeblue.com
www.shapeblue.com
Amadeus House, Floral Street, London WC2E 9DPUK
@shapeblue
-----Original Message-----
From: Ivan Kudryavtsev <ku...@bw-sw.com>
Sent: 23 October 2018 14:55
To: dev <de...@cloudstack.apache.org>
Subject: Re: KVM CloudStack Agent Hacking proposal
Hello, Paul. You have implemented the second part of the proposal, which is related to Qemu hook. Unfortunately, Qemu hooks are not always the right place to implement features. I would like it to be, but it's not because of:
https://www.libvirt.org/hooks.html#recursive
In my case, I even implemented a standalone unix-socket server which I have called from the hook without awaiting and which, in turn, forked the process to interact with libvirt, but it still causes the deadlock because of CS KVM agent and security_groups.py still does their interaction and it leads to a deadlock. So, hooks are not enough, but important to be. It's cool that current agent implementation already includes hooks - less work to do)
So, my proposal is to inject the capability into CS KVM agent. If the design I introduced in the first e-mail is OK, we can implement it.
> I would like is to introduce a more generic approach, so the
administrator can specify additional scripts in the
> agent.properties, which will be called the same way "security_groups.py"
called.
> custom.vm.start=/path/to/script1,path/to.script2
> custom.vm.stop=/path/to/script3,path/to.script4
Thank you for your time and opinions.
вт, 23 окт. 2018 г. в 2:56, Paul Angus <pa...@shapeblue.com>:
> Hi Ivan,
>
> I think that this may already have been added in 4.12 by ShapeBlue
>
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+hook+script
> +include
>
> if nothing else it sounds like you want to build upon this rather than
> rewrite it.
>
>
>
>
> paul.angus@shapeblue.com
> www.shapeblue.com
> Amadeus House, Floral Street, London WC2E 9DPUK @shapeblue
>
>
>
>
> -----Original Message-----
> From: Wido den Hollander <wi...@widodh.nl>
> Sent: 23 October 2018 07:46
> To: dev@cloudstack.apache.org
> Subject: Re: KVM CloudStack Agent Hacking proposal
>
>
>
> On 10/22/18 8:02 PM, Ivan Kudryavtsev wrote:
> > Hello, Devs.
> >
> > I would like to introduce a feature and decided to consult with you
> > about its design before implementation. The feature is connected with
> > KVM CloudStack agent. We have found it beneficial to be able to launch
> > custom scripts upon VM start/stop. It can be done using Qemu hook but
> > it has several drawbacks:
> > - the hook is deployed by CS and adding additional lines into it leads
> > to extra efforts when ACS package is updated.
> > - it leads to deadlocks as you cannot effectively and easy to
> > communicate with libvirt from hook even with "fork & exec" because
> > security_groups.py and agent also participate and as a result it causes
> deadlocks.
> >
> > Now, in the code, we have a call for "security_groups.py":
> >
> > Start:
> > https://github.com/apache/cloudstack/blob/65f31f1a9fbc1c20cd752d80a7e1
> > 117efc0248a5/plugins/hypervisors/kvm/src/main/java/com/cloud/hyperviso
> > r/kvm/resource/wrapper/LibvirtStartCommandWrapper.java#L103
> >
> > Stop:
> > https://github.com/apache/cloudstack/blob/65f31f1a9fbc1c20cd752d80a7e1
> > 117efc0248a5/plugins/hypervisors/kvm/src/main/java/com/cloud/hyperviso
> > r/kvm/resource/wrapper/LibvirtStopCommandWrapper.java#L88
> >
> > I would like is to introduce a more generic approach, so the
> > administrator can specify additional scripts in the agent.properties,
> > which will be called the same way "security_groups.py" called.
> >
> > custom.vm.start=/path/to/script1,path/to.script2
> > custom.vm.stop=/path/to/script3,path/to.script4
> >
> > So, this feature will help users to do custom hotplug mechanisms. E.g.
> > we have such implementation which adds per-account VXLAN as a hotplug
> > ethernet device. So, even for a Basic Zone, every VM gets automatic
> > second NIC which helps to build a private network for an account.
> >
> > Currently, we do the job thru adding lines into security_groups.py,
> > which is not a good approach, especially for end users who don't want
> > to hack the system.
> >
> > Also, I'm thinking about changing /etc/libvirt/hooks/qemu the same
> > way, so it was just an entry point to /etc/libvirt/hooks/qemu.d/*
> located scripts.
> >
> > Let me know about this feature proposal and if its design is good, we
> > start developing it.
> >
>
> Seems like a good thing! It adds flexibility to the VM.
>
> How are you planning on getting things like the VM name and other details
> to the scripts?
>
> Wido
>
> > Have a good day.
> >
>
--
With best regards, Ivan Kudryavtsev
Bitworks LLC
Cell RU: +7-923-414-1515
Cell USA: +1-201-257-1512
WWW: http://bitworks.software/ <http://bw-sw.com/>
Re: KVM CloudStack Agent Hacking proposal
Posted by Ivan Kudryavtsev <ku...@bw-sw.com>.
Hello, Paul. You have implemented the second part of the proposal, which is
related to Qemu hook. Unfortunately, Qemu hooks are not always the right
place to implement features. I would like it to be, but it's not because of:
https://www.libvirt.org/hooks.html#recursive
In my case, I even implemented a standalone unix-socket server which I have
called from the hook without awaiting and which, in turn, forked the
process to interact with libvirt, but it still causes the deadlock because
of CS KVM agent and security_groups.py still does their interaction and it
leads to a deadlock. So, hooks are not enough, but important to be. It's
cool that current agent implementation already includes hooks - less work
to do)
So, my proposal is to inject the capability into CS KVM agent. If the
design I introduced in the first e-mail is OK, we can implement it.
> I would like is to introduce a more generic approach, so the
administrator can specify additional scripts in the
> agent.properties, which will be called the same way "security_groups.py"
called.
> custom.vm.start=/path/to/script1,path/to.script2
> custom.vm.stop=/path/to/script3,path/to.script4
Thank you for your time and opinions.
вт, 23 окт. 2018 г. в 2:56, Paul Angus <pa...@shapeblue.com>:
> Hi Ivan,
>
> I think that this may already have been added in 4.12 by ShapeBlue
>
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+hook+script+include
>
> if nothing else it sounds like you want to build upon this rather than
> rewrite it.
>
>
>
>
> paul.angus@shapeblue.com
> www.shapeblue.com
> Amadeus House, Floral Street, London WC2E 9DPUK
> @shapeblue
>
>
>
>
> -----Original Message-----
> From: Wido den Hollander <wi...@widodh.nl>
> Sent: 23 October 2018 07:46
> To: dev@cloudstack.apache.org
> Subject: Re: KVM CloudStack Agent Hacking proposal
>
>
>
> On 10/22/18 8:02 PM, Ivan Kudryavtsev wrote:
> > Hello, Devs.
> >
> > I would like to introduce a feature and decided to consult with you
> > about its design before implementation. The feature is connected with
> > KVM CloudStack agent. We have found it beneficial to be able to launch
> > custom scripts upon VM start/stop. It can be done using Qemu hook but
> > it has several drawbacks:
> > - the hook is deployed by CS and adding additional lines into it leads
> > to extra efforts when ACS package is updated.
> > - it leads to deadlocks as you cannot effectively and easy to
> > communicate with libvirt from hook even with "fork & exec" because
> > security_groups.py and agent also participate and as a result it causes
> deadlocks.
> >
> > Now, in the code, we have a call for "security_groups.py":
> >
> > Start:
> > https://github.com/apache/cloudstack/blob/65f31f1a9fbc1c20cd752d80a7e1
> > 117efc0248a5/plugins/hypervisors/kvm/src/main/java/com/cloud/hyperviso
> > r/kvm/resource/wrapper/LibvirtStartCommandWrapper.java#L103
> >
> > Stop:
> > https://github.com/apache/cloudstack/blob/65f31f1a9fbc1c20cd752d80a7e1
> > 117efc0248a5/plugins/hypervisors/kvm/src/main/java/com/cloud/hyperviso
> > r/kvm/resource/wrapper/LibvirtStopCommandWrapper.java#L88
> >
> > I would like is to introduce a more generic approach, so the
> > administrator can specify additional scripts in the agent.properties,
> > which will be called the same way "security_groups.py" called.
> >
> > custom.vm.start=/path/to/script1,path/to.script2
> > custom.vm.stop=/path/to/script3,path/to.script4
> >
> > So, this feature will help users to do custom hotplug mechanisms. E.g.
> > we have such implementation which adds per-account VXLAN as a hotplug
> > ethernet device. So, even for a Basic Zone, every VM gets automatic
> > second NIC which helps to build a private network for an account.
> >
> > Currently, we do the job thru adding lines into security_groups.py,
> > which is not a good approach, especially for end users who don't want
> > to hack the system.
> >
> > Also, I'm thinking about changing /etc/libvirt/hooks/qemu the same
> > way, so it was just an entry point to /etc/libvirt/hooks/qemu.d/*
> located scripts.
> >
> > Let me know about this feature proposal and if its design is good, we
> > start developing it.
> >
>
> Seems like a good thing! It adds flexibility to the VM.
>
> How are you planning on getting things like the VM name and other details
> to the scripts?
>
> Wido
>
> > Have a good day.
> >
>
--
With best regards, Ivan Kudryavtsev
Bitworks LLC
Cell RU: +7-923-414-1515
Cell USA: +1-201-257-1512
WWW: http://bitworks.software/ <http://bw-sw.com/>
RE: KVM CloudStack Agent Hacking proposal
Posted by Paul Angus <pa...@shapeblue.com>.
Hi Ivan,
I think that this may already have been added in 4.12 by ShapeBlue
https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+hook+script+include
if nothing else it sounds like you want to build upon this rather than rewrite it.
paul.angus@shapeblue.com
www.shapeblue.com
Amadeus House, Floral Street, London WC2E 9DPUK
@shapeblue
-----Original Message-----
From: Wido den Hollander <wi...@widodh.nl>
Sent: 23 October 2018 07:46
To: dev@cloudstack.apache.org
Subject: Re: KVM CloudStack Agent Hacking proposal
On 10/22/18 8:02 PM, Ivan Kudryavtsev wrote:
> Hello, Devs.
>
> I would like to introduce a feature and decided to consult with you
> about its design before implementation. The feature is connected with
> KVM CloudStack agent. We have found it beneficial to be able to launch
> custom scripts upon VM start/stop. It can be done using Qemu hook but
> it has several drawbacks:
> - the hook is deployed by CS and adding additional lines into it leads
> to extra efforts when ACS package is updated.
> - it leads to deadlocks as you cannot effectively and easy to
> communicate with libvirt from hook even with "fork & exec" because
> security_groups.py and agent also participate and as a result it causes deadlocks.
>
> Now, in the code, we have a call for "security_groups.py":
>
> Start:
> https://github.com/apache/cloudstack/blob/65f31f1a9fbc1c20cd752d80a7e1
> 117efc0248a5/plugins/hypervisors/kvm/src/main/java/com/cloud/hyperviso
> r/kvm/resource/wrapper/LibvirtStartCommandWrapper.java#L103
>
> Stop:
> https://github.com/apache/cloudstack/blob/65f31f1a9fbc1c20cd752d80a7e1
> 117efc0248a5/plugins/hypervisors/kvm/src/main/java/com/cloud/hyperviso
> r/kvm/resource/wrapper/LibvirtStopCommandWrapper.java#L88
>
> I would like is to introduce a more generic approach, so the
> administrator can specify additional scripts in the agent.properties,
> which will be called the same way "security_groups.py" called.
>
> custom.vm.start=/path/to/script1,path/to.script2
> custom.vm.stop=/path/to/script3,path/to.script4
>
> So, this feature will help users to do custom hotplug mechanisms. E.g.
> we have such implementation which adds per-account VXLAN as a hotplug
> ethernet device. So, even for a Basic Zone, every VM gets automatic
> second NIC which helps to build a private network for an account.
>
> Currently, we do the job thru adding lines into security_groups.py,
> which is not a good approach, especially for end users who don't want
> to hack the system.
>
> Also, I'm thinking about changing /etc/libvirt/hooks/qemu the same
> way, so it was just an entry point to /etc/libvirt/hooks/qemu.d/* located scripts.
>
> Let me know about this feature proposal and if its design is good, we
> start developing it.
>
Seems like a good thing! It adds flexibility to the VM.
How are you planning on getting things like the VM name and other details to the scripts?
Wido
> Have a good day.
>
Re: KVM CloudStack Agent Hacking proposal
Posted by Ivan Kudryavtsev <ku...@bw-sw.com>.
Wido,
> How are you planning on getting things like the VM name and other
details to the scripts?
Agent passes it right now into SGs, so I'm thinking about the same way:
https://github.com/apache/cloudstack/blob/65f31f1a9fbc1c20cd752d80a7e1117efc0248a5/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtStartCommandWrapper.java#L103
As about VM details, I think, the script can get them from libvirt if
necessary or from /var/run/libvirt/qemu it's up to the user how to get what
he/she needs. What I care about is about avoiding the deadlocks.
вт, 23 окт. 2018 г. в 2:46, Wido den Hollander <wi...@widodh.nl>:
>
>
> On 10/22/18 8:02 PM, Ivan Kudryavtsev wrote:
> > Hello, Devs.
> >
> > I would like to introduce a feature and decided to consult with you about
> > its design before implementation. The feature is connected with KVM
> > CloudStack agent. We have found it beneficial to be able to launch custom
> > scripts upon VM start/stop. It can be done using Qemu hook but it has
> > several drawbacks:
> > - the hook is deployed by CS and adding additional lines into it leads to
> > extra efforts when ACS package is updated.
> > - it leads to deadlocks as you cannot effectively and easy to communicate
> > with libvirt from hook even with "fork & exec" because security_groups.py
> > and agent also participate and as a result it causes deadlocks.
> >
> > Now, in the code, we have a call for "security_groups.py":
> >
> > Start:
> >
> https://github.com/apache/cloudstack/blob/65f31f1a9fbc1c20cd752d80a7e1117efc0248a5/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtStartCommandWrapper.java#L103
> >
> > Stop:
> >
> https://github.com/apache/cloudstack/blob/65f31f1a9fbc1c20cd752d80a7e1117efc0248a5/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtStopCommandWrapper.java#L88
> >
> > I would like is to introduce a more generic approach, so the
> administrator
> > can specify additional scripts in the agent.properties, which will be
> > called the same way "security_groups.py" called.
> >
> > custom.vm.start=/path/to/script1,path/to.script2
> > custom.vm.stop=/path/to/script3,path/to.script4
> >
> > So, this feature will help users to do custom hotplug mechanisms. E.g. we
> > have such implementation which adds per-account VXLAN as a hotplug
> ethernet
> > device. So, even for a Basic Zone, every VM gets automatic second NIC
> which
> > helps to build a private network for an account.
> >
> > Currently, we do the job thru adding lines into security_groups.py, which
> > is not a good approach, especially for end users who don't want to hack
> the
> > system.
> >
> > Also, I'm thinking about changing /etc/libvirt/hooks/qemu the same way,
> so
> > it was just an entry point to /etc/libvirt/hooks/qemu.d/* located
> scripts.
> >
> > Let me know about this feature proposal and if its design is good, we
> start
> > developing it.
> >
>
> Seems like a good thing! It adds flexibility to the VM.
>
> How are you planning on getting things like the VM name and other
> details to the scripts?
>
> Wido
>
> > Have a good day.
> >
>
--
With best regards, Ivan Kudryavtsev
Bitworks LLC
Cell RU: +7-923-414-1515
Cell USA: +1-201-257-1512
WWW: http://bitworks.software/ <http://bw-sw.com/>
Re: KVM CloudStack Agent Hacking proposal
Posted by Wido den Hollander <wi...@widodh.nl>.
On 10/22/18 8:02 PM, Ivan Kudryavtsev wrote:
> Hello, Devs.
>
> I would like to introduce a feature and decided to consult with you about
> its design before implementation. The feature is connected with KVM
> CloudStack agent. We have found it beneficial to be able to launch custom
> scripts upon VM start/stop. It can be done using Qemu hook but it has
> several drawbacks:
> - the hook is deployed by CS and adding additional lines into it leads to
> extra efforts when ACS package is updated.
> - it leads to deadlocks as you cannot effectively and easy to communicate
> with libvirt from hook even with "fork & exec" because security_groups.py
> and agent also participate and as a result it causes deadlocks.
>
> Now, in the code, we have a call for "security_groups.py":
>
> Start:
> https://github.com/apache/cloudstack/blob/65f31f1a9fbc1c20cd752d80a7e1117efc0248a5/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtStartCommandWrapper.java#L103
>
> Stop:
> https://github.com/apache/cloudstack/blob/65f31f1a9fbc1c20cd752d80a7e1117efc0248a5/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtStopCommandWrapper.java#L88
>
> I would like is to introduce a more generic approach, so the administrator
> can specify additional scripts in the agent.properties, which will be
> called the same way "security_groups.py" called.
>
> custom.vm.start=/path/to/script1,path/to.script2
> custom.vm.stop=/path/to/script3,path/to.script4
>
> So, this feature will help users to do custom hotplug mechanisms. E.g. we
> have such implementation which adds per-account VXLAN as a hotplug ethernet
> device. So, even for a Basic Zone, every VM gets automatic second NIC which
> helps to build a private network for an account.
>
> Currently, we do the job thru adding lines into security_groups.py, which
> is not a good approach, especially for end users who don't want to hack the
> system.
>
> Also, I'm thinking about changing /etc/libvirt/hooks/qemu the same way, so
> it was just an entry point to /etc/libvirt/hooks/qemu.d/* located scripts.
>
> Let me know about this feature proposal and if its design is good, we start
> developing it.
>
Seems like a good thing! It adds flexibility to the VM.
How are you planning on getting things like the VM name and other
details to the scripts?
Wido
> Have a good day.
>