You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stratos.apache.org by Imesh Gunaratne <im...@apache.org> on 2015/02/05 13:55:05 UTC
[Discuss] Writing Cartridge Agent Extensions in Python
Hi Devs,
Sometime back we had a discussion on introducing a python plugin framework
for implementing extensions for Python Cartridge Agent (PCA) in python.
However we could not take any actions on it.
Currently we have bash scripts as extensions however if we could introduce
a python extension layer in between that would make it much easier to
implement extension logic.
IMO we could wrap existing extension bashscripts with a set of python
scripts and invoke those by the agent. If a cartridge would like to write
extensions in python it could use those. If some other cartridge would
prefer to use bash that's also possible. We should be able to do this with
less effort. WDYT?
Thanks
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PMC Member, Apache Stratos
Re: [Discuss] Writing Cartridge Agent Extensions in Python
Posted by Chamila De Alwis <ch...@wso2.com>.
Hi Imesh,
On Thu, Feb 19, 2015 at 2:47 PM, Imesh Gunaratne <im...@apache.org> wrote:
> I think we can introduce a packaging model for plugins using zip. PCA can
> provide a folder to copy plugin zip files (a deploy folder: plugins/deploy)
> and at the startup those can be extracted to a different folder
> (plugins/runtime)
+1. This will be a great addition. I will look in to this soon and send a
PR.
Regards,
Chamila de Alwis
Software Engineer | WSO2 | +94772207163
Blog: code.chamiladealwis.com
Re: [Discuss] Writing Cartridge Agent Extensions in Python
Posted by Imesh Gunaratne <im...@apache.org>.
Hi Chamila,
It looks good! I have now merged this pull request to master branch.
I think we can introduce a packaging model for plugins using zip. PCA can
provide a folder to copy plugin zip files (a deploy folder: plugins/deploy)
and at the startup those can be extracted to a different folder
(plugins/runtime). WDYT?
Thanks
On Thu, Feb 19, 2015 at 10:35 AM, Chamila De Alwis <ch...@wso2.com>
wrote:
> Hi Imesh,
>
> I've opened a PR for this feature at [1]. Please review and merge.
>
> [1] - https://github.com/apache/stratos/pull/234
>
>
> Regards,
> Chamila de Alwis
> Software Engineer | WSO2 | +94772207163
> Blog: code.chamiladealwis.com
>
>
>
> On Sat, Feb 14, 2015 at 10:02 PM, Chamila De Alwis <ch...@wso2.com>
> wrote:
>
>> Hi Imesh,
>>
>> On Sat, Feb 14, 2015 at 9:37 PM, Imesh Gunaratne <im...@apache.org>
>> wrote:
>>
>>> Will it be possible to design the plugin contract to be able to listen
>>> to multiple events?
>>>
>>
>> Yes, it is possible to modify the plugin description to define several
>> events for a single plugin. For an example, the list of events to map a
>> particular plugin can be defined as a comma separated value in the
>> description. I will make the changes necessary.
>>
>>
>> Regards,
>> Chamila de Alwis
>> Software Engineer | WSO2 | +94772207163
>> Blog: code.chamiladealwis.com
>>
>>
>>
>
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PMC Member, Apache Stratos
Re: [Discuss] Writing Cartridge Agent Extensions in Python
Posted by Chamila De Alwis <ch...@wso2.com>.
Hi Imesh,
I've opened a PR for this feature at [1]. Please review and merge.
[1] - https://github.com/apache/stratos/pull/234
Regards,
Chamila de Alwis
Software Engineer | WSO2 | +94772207163
Blog: code.chamiladealwis.com
On Sat, Feb 14, 2015 at 10:02 PM, Chamila De Alwis <ch...@wso2.com>
wrote:
> Hi Imesh,
>
> On Sat, Feb 14, 2015 at 9:37 PM, Imesh Gunaratne <im...@apache.org> wrote:
>
>> Will it be possible to design the plugin contract to be able to listen
>> to multiple events?
>>
>
> Yes, it is possible to modify the plugin description to define several
> events for a single plugin. For an example, the list of events to map a
> particular plugin can be defined as a comma separated value in the
> description. I will make the changes necessary.
>
>
> Regards,
> Chamila de Alwis
> Software Engineer | WSO2 | +94772207163
> Blog: code.chamiladealwis.com
>
>
>
Re: [Discuss] Writing Cartridge Agent Extensions in Python
Posted by Chamila De Alwis <ch...@wso2.com>.
Hi Imesh,
On Sat, Feb 14, 2015 at 9:37 PM, Imesh Gunaratne <im...@apache.org> wrote:
> Will it be possible to design the plugin contract to be able to listen to
> multiple events?
>
Yes, it is possible to modify the plugin description to define several
events for a single plugin. For an example, the list of events to map a
particular plugin can be defined as a comma separated value in the
description. I will make the changes necessary.
Regards,
Chamila de Alwis
Software Engineer | WSO2 | +94772207163
Blog: code.chamiladealwis.com
Re: [Discuss] Writing Cartridge Agent Extensions in Python
Posted by Imesh Gunaratne <im...@apache.org>.
Great work Chamila!
Will it be possible to design the plugin contract to be able to listen to
multiple events? If so plugins can implement features in PCA rather than
being restricting to single event actions. For an example if a cartridge
needs to support a certain clustering technology, we could provide a
clustering plugin for that particular clustering framework.
On Sat, Feb 14, 2015 at 9:10 PM, Chamila De Alwis <ch...@wso2.com> wrote:
> Hi,
>
> I have now implemented a basic plugin system where *<PCA_HOME>/plugins*
> folder can be used to drop plugins to be picked up by the PCA. For this
> I've used Yapsy[1] plugin framework. As Akila mentioned in a previous
> thread, Yapsy is both simple and effective in enabling a plugin structure
> in Python. It is BSD licensed so there is no conflict in licensing for
> incorporating it, if the need arises.
>
> Implementing a plugin for the PCA is simple. The plugin contract is
> defined in *<PCA_HOME>/plugins/contracts.py* module. To write a plugin
> *contracts.ICartridgeAgentPlugin* has to be extended. Additionally a
> plugin descriptor (with yapsy-plugin extension) has to be included which
> will describe and point to the plugin module file, when queried by the
> Yapsy framework. To map a plugin for a certain event, the event name has to
> be specified in the Description field. By this approach, it is possible to
> map more than one plugin to a single event. Each plugin is run in a
> separate thread, therefore, it will not be blocking the main execution.
>
> A plugin gets a dictionary with the payload parameters and any additional
> values included for specific events. This is more effective than the
> environment variables approach we had for bash scripts.
>
> I have included a test plugin which is mapped to the
> MemberInitializedEvent as a POC. All it does is to output the values
> provided in the dictionary.
>
> I'm hoping to extend this structure to accept artifact management plugins
> (the contract for which would include checkout() and push() methods) and
> health statistics reader plugins. I will send a PR soon with the
> modifications.
>
> TestPlugin.yapsy-plugin
>
> [Core]
> Name = Test plugin for MemberInitializedEvent
> Module = TestPlugin
>
> [Documentation]
> Description = MemberInitializedEvent
> Author = Op1
> Version = 0.1
> Website = stratos.apache.org
>
>
> TestPlugin.py
>
>
> from plugins.contracts import ICartridgeAgentPlugin
>
>
> class TestPlugin(ICartridgeAgentPlugin):
>
> def run_plugin(self, values, log):
> log.debug("Running test plugin")
> for key, value in values.iteritems():
> log.debug("%s => %s" % (key, value))
>
>
> PCA folder structure
>
>
>
>
> [1] - http://yapsy.sourceforge.net/
>
>
> Regards,
> Chamila de Alwis
> Software Engineer | WSO2 | +94772207163
> Blog: code.chamiladealwis.com
>
>
>
> On Thu, Feb 5, 2015 at 11:56 PM, Chamila De Alwis <ch...@wso2.com>
> wrote:
>
>> Hi,
>>
>> The current extension setup is as follows. A set of bash scripts are
>> available in the <PCA_HOME>/extensions. The file name of the extension that
>> should be executed for each event is defined in the agent configuration
>> (agent.conf in PCA, stratos.sh in JCA). The event handler executes these
>> bash scripts as new shell commands when the respective events are received.
>>
>> We can introduce a layer between the bash scripts and the python code
>> that calls them. The names of the python scripts corresponding to each
>> event can be defined as before.
>>
>> Additionally we can extend this behavior to the Git artifact management
>> component of the PCA. The contract that the artifact management component
>> should adhere to is "checkout" and "push". We can separate the current
>> python code relating to these two operations in to two extension scripts.
>> This will also address the platform dependence issue we are experiencing
>> now with the git binary. I will start working on this.
>>
>> What do you think?
>>
>>
>> Regards,
>> Chamila de Alwis
>> Software Engineer | WSO2 | +94772207163
>> Blog: code.chamiladealwis.com
>>
>>
>>
>> On Thu, Feb 5, 2015 at 6:57 PM, Gayan Gunarathne <ga...@wso2.com> wrote:
>>
>>> Also I guess we can ship the default set of extensions for both python
>>> and bash with python cartridge agent. With the user requirement, user can
>>> introduce own set of extensions as their choice.
>>>
>>> Thanks,
>>> Gayan
>>>
>>> On Thu, Feb 5, 2015 at 6:51 PM, Gayan Gunarathne <ga...@wso2.com>
>>> wrote:
>>>
>>>> +1 to have to both python extensions and bash extensions .If we can
>>>> introduce the python extension layer ,it will give more options for
>>>> the user.
>>>>
>>>> Thanks,
>>>> Gayan
>>>>
>>>>
>>>>
>>>> On Thu, Feb 5, 2015 at 6:25 PM, Imesh Gunaratne <im...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Devs,
>>>>>
>>>>> Sometime back we had a discussion on introducing a python plugin
>>>>> framework for implementing extensions for Python Cartridge Agent (PCA) in
>>>>> python. However we could not take any actions on it.
>>>>>
>>>>> Currently we have bash scripts as extensions however if we could
>>>>> introduce a python extension layer in between that would make it much
>>>>> easier to implement extension logic.
>>>>>
>>>>> IMO we could wrap existing extension bashscripts with a set of python
>>>>> scripts and invoke those by the agent. If a cartridge would like to write
>>>>> extensions in python it could use those. If some other cartridge would
>>>>> prefer to use bash that's also possible. We should be able to do this with
>>>>> less effort. WDYT?
>>>>>
>>>>> Thanks
>>>>>
>>>>> --
>>>>> Imesh Gunaratne
>>>>>
>>>>> Technical Lead, WSO2
>>>>> Committer & PMC Member, Apache Stratos
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Gayan Gunarathne
>>>> Technical Lead
>>>> WSO2 Inc. (http://wso2.com)
>>>> email : gayang@wso2.com | mobile : +94 766819985
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Gayan Gunarathne
>>> Technical Lead
>>> WSO2 Inc. (http://wso2.com)
>>> email : gayang@wso2.com | mobile : +94 766819985
>>>
>>>
>>
>>
>
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PMC Member, Apache Stratos
Re: [Discuss] Writing Cartridge Agent Extensions in Python
Posted by Chamila De Alwis <ch...@wso2.com>.
Hi,
I have now implemented a basic plugin system where *<PCA_HOME>/plugins*
folder can be used to drop plugins to be picked up by the PCA. For this
I've used Yapsy[1] plugin framework. As Akila mentioned in a previous
thread, Yapsy is both simple and effective in enabling a plugin structure
in Python. It is BSD licensed so there is no conflict in licensing for
incorporating it, if the need arises.
Implementing a plugin for the PCA is simple. The plugin contract is defined
in *<PCA_HOME>/plugins/contracts.py* module. To write a plugin
*contracts.ICartridgeAgentPlugin* has to be extended. Additionally a plugin
descriptor (with yapsy-plugin extension) has to be included which will
describe and point to the plugin module file, when queried by the Yapsy
framework. To map a plugin for a certain event, the event name has to be
specified in the Description field. By this approach, it is possible to map
more than one plugin to a single event. Each plugin is run in a separate
thread, therefore, it will not be blocking the main execution.
A plugin gets a dictionary with the payload parameters and any additional
values included for specific events. This is more effective than the
environment variables approach we had for bash scripts.
I have included a test plugin which is mapped to the MemberInitializedEvent
as a POC. All it does is to output the values provided in the dictionary.
I'm hoping to extend this structure to accept artifact management plugins
(the contract for which would include checkout() and push() methods) and
health statistics reader plugins. I will send a PR soon with the
modifications.
TestPlugin.yapsy-plugin
[Core]
Name = Test plugin for MemberInitializedEvent
Module = TestPlugin
[Documentation]
Description = MemberInitializedEvent
Author = Op1
Version = 0.1
Website = stratos.apache.org
TestPlugin.py
from plugins.contracts import ICartridgeAgentPlugin
class TestPlugin(ICartridgeAgentPlugin):
def run_plugin(self, values, log):
log.debug("Running test plugin")
for key, value in values.iteritems():
log.debug("%s => %s" % (key, value))
PCA folder structure
[1] - http://yapsy.sourceforge.net/
Regards,
Chamila de Alwis
Software Engineer | WSO2 | +94772207163
Blog: code.chamiladealwis.com
On Thu, Feb 5, 2015 at 11:56 PM, Chamila De Alwis <ch...@wso2.com> wrote:
> Hi,
>
> The current extension setup is as follows. A set of bash scripts are
> available in the <PCA_HOME>/extensions. The file name of the extension that
> should be executed for each event is defined in the agent configuration
> (agent.conf in PCA, stratos.sh in JCA). The event handler executes these
> bash scripts as new shell commands when the respective events are received.
>
> We can introduce a layer between the bash scripts and the python code that
> calls them. The names of the python scripts corresponding to each event can
> be defined as before.
>
> Additionally we can extend this behavior to the Git artifact management
> component of the PCA. The contract that the artifact management component
> should adhere to is "checkout" and "push". We can separate the current
> python code relating to these two operations in to two extension scripts.
> This will also address the platform dependence issue we are experiencing
> now with the git binary. I will start working on this.
>
> What do you think?
>
>
> Regards,
> Chamila de Alwis
> Software Engineer | WSO2 | +94772207163
> Blog: code.chamiladealwis.com
>
>
>
> On Thu, Feb 5, 2015 at 6:57 PM, Gayan Gunarathne <ga...@wso2.com> wrote:
>
>> Also I guess we can ship the default set of extensions for both python
>> and bash with python cartridge agent. With the user requirement, user can
>> introduce own set of extensions as their choice.
>>
>> Thanks,
>> Gayan
>>
>> On Thu, Feb 5, 2015 at 6:51 PM, Gayan Gunarathne <ga...@wso2.com> wrote:
>>
>>> +1 to have to both python extensions and bash extensions .If we can
>>> introduce the python extension layer ,it will give more options for the
>>> user.
>>>
>>> Thanks,
>>> Gayan
>>>
>>>
>>>
>>> On Thu, Feb 5, 2015 at 6:25 PM, Imesh Gunaratne <im...@apache.org>
>>> wrote:
>>>
>>>> Hi Devs,
>>>>
>>>> Sometime back we had a discussion on introducing a python plugin
>>>> framework for implementing extensions for Python Cartridge Agent (PCA) in
>>>> python. However we could not take any actions on it.
>>>>
>>>> Currently we have bash scripts as extensions however if we could
>>>> introduce a python extension layer in between that would make it much
>>>> easier to implement extension logic.
>>>>
>>>> IMO we could wrap existing extension bashscripts with a set of python
>>>> scripts and invoke those by the agent. If a cartridge would like to write
>>>> extensions in python it could use those. If some other cartridge would
>>>> prefer to use bash that's also possible. We should be able to do this with
>>>> less effort. WDYT?
>>>>
>>>> Thanks
>>>>
>>>> --
>>>> Imesh Gunaratne
>>>>
>>>> Technical Lead, WSO2
>>>> Committer & PMC Member, Apache Stratos
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Gayan Gunarathne
>>> Technical Lead
>>> WSO2 Inc. (http://wso2.com)
>>> email : gayang@wso2.com | mobile : +94 766819985
>>>
>>>
>>
>>
>>
>> --
>>
>> Gayan Gunarathne
>> Technical Lead
>> WSO2 Inc. (http://wso2.com)
>> email : gayang@wso2.com | mobile : +94 766819985
>>
>>
>
>
Re: [Discuss] Writing Cartridge Agent Extensions in Python
Posted by Chamila De Alwis <ch...@wso2.com>.
Hi,
The current extension setup is as follows. A set of bash scripts are
available in the <PCA_HOME>/extensions. The file name of the extension that
should be executed for each event is defined in the agent configuration
(agent.conf in PCA, stratos.sh in JCA). The event handler executes these
bash scripts as new shell commands when the respective events are received.
We can introduce a layer between the bash scripts and the python code that
calls them. The names of the python scripts corresponding to each event can
be defined as before.
Additionally we can extend this behavior to the Git artifact management
component of the PCA. The contract that the artifact management component
should adhere to is "checkout" and "push". We can separate the current
python code relating to these two operations in to two extension scripts.
This will also address the platform dependence issue we are experiencing
now with the git binary. I will start working on this.
What do you think?
Regards,
Chamila de Alwis
Software Engineer | WSO2 | +94772207163
Blog: code.chamiladealwis.com
On Thu, Feb 5, 2015 at 6:57 PM, Gayan Gunarathne <ga...@wso2.com> wrote:
> Also I guess we can ship the default set of extensions for both python and
> bash with python cartridge agent. With the user requirement, user can
> introduce own set of extensions as their choice.
>
> Thanks,
> Gayan
>
> On Thu, Feb 5, 2015 at 6:51 PM, Gayan Gunarathne <ga...@wso2.com> wrote:
>
>> +1 to have to both python extensions and bash extensions .If we can
>> introduce the python extension layer ,it will give more options for the
>> user.
>>
>> Thanks,
>> Gayan
>>
>>
>>
>> On Thu, Feb 5, 2015 at 6:25 PM, Imesh Gunaratne <im...@apache.org> wrote:
>>
>>> Hi Devs,
>>>
>>> Sometime back we had a discussion on introducing a python plugin
>>> framework for implementing extensions for Python Cartridge Agent (PCA) in
>>> python. However we could not take any actions on it.
>>>
>>> Currently we have bash scripts as extensions however if we could
>>> introduce a python extension layer in between that would make it much
>>> easier to implement extension logic.
>>>
>>> IMO we could wrap existing extension bashscripts with a set of python
>>> scripts and invoke those by the agent. If a cartridge would like to write
>>> extensions in python it could use those. If some other cartridge would
>>> prefer to use bash that's also possible. We should be able to do this with
>>> less effort. WDYT?
>>>
>>> Thanks
>>>
>>> --
>>> Imesh Gunaratne
>>>
>>> Technical Lead, WSO2
>>> Committer & PMC Member, Apache Stratos
>>>
>>
>>
>>
>> --
>>
>> Gayan Gunarathne
>> Technical Lead
>> WSO2 Inc. (http://wso2.com)
>> email : gayang@wso2.com | mobile : +94 766819985
>>
>>
>
>
>
> --
>
> Gayan Gunarathne
> Technical Lead
> WSO2 Inc. (http://wso2.com)
> email : gayang@wso2.com | mobile : +94 766819985
>
>
Re: [Discuss] Writing Cartridge Agent Extensions in Python
Posted by Gayan Gunarathne <ga...@wso2.com>.
Also I guess we can ship the default set of extensions for both python and
bash with python cartridge agent. With the user requirement, user can
introduce own set of extensions as their choice.
Thanks,
Gayan
On Thu, Feb 5, 2015 at 6:51 PM, Gayan Gunarathne <ga...@wso2.com> wrote:
> +1 to have to both python extensions and bash extensions .If we can
> introduce the python extension layer ,it will give more options for the
> user.
>
> Thanks,
> Gayan
>
>
>
> On Thu, Feb 5, 2015 at 6:25 PM, Imesh Gunaratne <im...@apache.org> wrote:
>
>> Hi Devs,
>>
>> Sometime back we had a discussion on introducing a python plugin
>> framework for implementing extensions for Python Cartridge Agent (PCA) in
>> python. However we could not take any actions on it.
>>
>> Currently we have bash scripts as extensions however if we could
>> introduce a python extension layer in between that would make it much
>> easier to implement extension logic.
>>
>> IMO we could wrap existing extension bashscripts with a set of python
>> scripts and invoke those by the agent. If a cartridge would like to write
>> extensions in python it could use those. If some other cartridge would
>> prefer to use bash that's also possible. We should be able to do this with
>> less effort. WDYT?
>>
>> Thanks
>>
>> --
>> Imesh Gunaratne
>>
>> Technical Lead, WSO2
>> Committer & PMC Member, Apache Stratos
>>
>
>
>
> --
>
> Gayan Gunarathne
> Technical Lead
> WSO2 Inc. (http://wso2.com)
> email : gayang@wso2.com | mobile : +94 766819985
>
>
--
Gayan Gunarathne
Technical Lead
WSO2 Inc. (http://wso2.com)
email : gayang@wso2.com | mobile : +94 766819985
Re: [Discuss] Writing Cartridge Agent Extensions in Python
Posted by Gayan Gunarathne <ga...@wso2.com>.
+1 to have to both python extensions and bash extensions .If we can
introduce the python extension layer ,it will give more options for the
user.
Thanks,
Gayan
On Thu, Feb 5, 2015 at 6:25 PM, Imesh Gunaratne <im...@apache.org> wrote:
> Hi Devs,
>
> Sometime back we had a discussion on introducing a python plugin framework
> for implementing extensions for Python Cartridge Agent (PCA) in python.
> However we could not take any actions on it.
>
> Currently we have bash scripts as extensions however if we could introduce
> a python extension layer in between that would make it much easier to
> implement extension logic.
>
> IMO we could wrap existing extension bashscripts with a set of python
> scripts and invoke those by the agent. If a cartridge would like to write
> extensions in python it could use those. If some other cartridge would
> prefer to use bash that's also possible. We should be able to do this with
> less effort. WDYT?
>
> Thanks
>
> --
> Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>
--
Gayan Gunarathne
Technical Lead
WSO2 Inc. (http://wso2.com)
email : gayang@wso2.com | mobile : +94 766819985