[DISCUSS] OSGi framework for plugins and more?

[Hugo Trippaers <HT...@schubergphilis.com>]

Hey  all,

With the recent discussions on refactoring CloudStack and the working going into javelin I would like to discuss using OSGi. The background is that I have been struggling with ideas on how to setup a plugin system for CloudStack that would allow plugins to be separate entities which can be release independently from CloudStack core. Mainly to deal with the current non-asf components but for future expansion as well.

While at ApacheConEU I had the chance to discuss these ideas with one of our mentors after his talk about OSGi. I'm pretty charmed by OSGi and the options it provides. It's a well thought out system that allow true modularity and pluggability. With the amount of support in the java industry it's a standard that feels very mature and a safe bet, one I would prefer to any homegrown plugin system. It supports versioning of components, strict separation of classes between components and all kind of funky features like hot-plugging and hot-replace. Using OSGi would mean that people can supply bundles with functionality which are maintained separately from the 'main code' without having to worry about how to integrate it with the core. Just putting the module in the right directory should be enough to get CloudStack to see and use the bundle. Upgrades happen the same way, new version of an authenticator, just replace the bundle and let the framework replace it with having to shutdown the server at all.

As we are discussing making CloudStack more modular, I would like to propose to start using OSGi for this. It is a bit of a learning curve to start with, but one we can get help with from our mentors. I'm already working on setting up a proof of concept for a plugin system using OSGi together with a colleague to show how it works, code is always better than words.

So what are your thoughts?

Cheers,

Hugo

Re: [DISCUSS] OSGi framework for plugins and more?

[Charles Moulliard <ch...@gmail.com>]

Hi Hugo,

As committer on Apache Karaf (multi-containers platform running top of
Apache Felix runtime), we can help you to leverage our expertise in that
topic. This is also something that I proposed this morning to develop the
cloudmonkey client using apache Karaf as we have done for jclouds api.

Regards,

Charles


On Mon, Nov 12, 2012 at 10:12 AM, Hugo Trippaers <
HTrippaers@schubergphilis.com> wrote:

> Hey  all,
>
> With the recent discussions on refactoring CloudStack and the working
> going into javelin I would like to discuss using OSGi. The background is
> that I have been struggling with ideas on how to setup a plugin system for
> CloudStack that would allow plugins to be separate entities which can be
> release independently from CloudStack core. Mainly to deal with the current
> non-asf components but for future expansion as well.
>
> While at ApacheConEU I had the chance to discuss these ideas with one of
> our mentors after his talk about OSGi. I'm pretty charmed by OSGi and the
> options it provides. It's a well thought out system that allow true
> modularity and pluggability. With the amount of support in the java
> industry it's a standard that feels very mature and a safe bet, one I would
> prefer to any homegrown plugin system. It supports versioning of
> components, strict separation of classes between components and all kind of
> funky features like hot-plugging and hot-replace. Using OSGi would mean
> that people can supply bundles with functionality which are maintained
> separately from the 'main code' without having to worry about how to
> integrate it with the core. Just putting the module in the right directory
> should be enough to get CloudStack to see and use the bundle. Upgrades
> happen the same way, new version of an authenticator, just replace the
> bundle and let the framework replace it with having to shutdown the server
> at all.
>
> As we are discussing making CloudStack more modular, I would like to
> propose to start using OSGi for this. It is a bit of a learning curve to
> start with, but one we can get help with from our mentors. I'm already
> working on setting up a proof of concept for a plugin system using OSGi
> together with a colleague to show how it works, code is always better than
> words.
>
> So what are your thoughts?
>
> Cheers,
>
> Hugo
>
>


-- 
Charles Moulliard
Apache Committer / Sr. Enterprise Architect (RedHat)
Twitter : @cmoulliard | Blog : http://cmoulliard.blogspot.com

RE: [DISCUSS] OSGi framework for plugins and more?

[Mohammad Nour El-Din <no...@gmail.com>]

hi

Sent from my Samsung Galaxy S3
Apologies for any typos
On Nov 15, 2012 6:52 PM, "Donal Lafferty" <do...@citrix.com> wrote:
>
> 1.  Is each service going to be a separate process?

well services being accessed using their rest interface can ease that for
sure but will not limit it

>
> 2.  Will the component model follow the current kernel / plugin strategy?

I am not the right one to answer that. anyone else would like to give
feedback abt this point ?

>
> DL
>
> -----Original Message-----
> From: Mohammad Nour El-Din [mailto:nour.mohammad@gmail.com]
> Sent: 15 November 2012 5:43 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: [DISCUSS] OSGi framework for plugins and more?
>
> sounds like a good plan. allow me to brief what has been said:
>
> 1- disaggregatio of ACS into a set of services which are accessed
restfully
> 2- the components of these services can be implemented as OSGi or at
least OSGi ready
>
> any missing points ? additional ones ? other insights ?
>
> Sent from my Samsung Galaxy S3
> Apologies for any typos
> On Nov 15, 2012 6:20 PM, "Alex Huang" <Al...@citrix.com> wrote:
> >
> >
> >
> > > -----Original Message-----
> > > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > > Sent: Thursday, November 15, 2012 9:15 AM
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: RE: [DISCUSS] OSGi framework for plugins and more?
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Mohammad Nour El-Din [mailto:nour.mohammad@gmail.com]
> > > > Sent: Wednesday, November 14, 2012 7:11 AM
> > > > To: cloudstack-dev@incubator.apache.org
> > > > Subject: Re: [DISCUSS] OSGi framework for plugins and more?
> > > >
> > > > Hi Alex...
> > > >
> > > >    Thanks for opening the discussion in the direction of what we
> > > > need
> to do
> > > > and how to do it, I thought no one will respond to my request :D
> > > >
> > > > On Wed, Nov 14, 2012 at 3:45 PM, Alex Huang
> > > > <Al...@citrix.com>
> > > > wrote:
> > > >
> > > > > Here's my two-bits on OSGi.  I actually started a thread like
> > > > > this sometime ago to which Mohammad reference.  I then did some
> > > > > research
> > > > on what
> > > > > OSGi can do.  The problems I looked at using OSGi for just isn't
> solvable
> > > > > by OSGi.
> > > > >
> > > > > For example,
> > > > > - How to contain fault within an availability zone.
> > > > > - How to do rolling upgrade and phase out the rolling upgrades
> > > > > over
> a
> > > span
> > > > > of days to deal with the time that it might take.
> > > > > - How to do database upgrades/downgrades for the plugins
> > > > > - How to scale different components differently
> > > > >
> > > > > OSGi does solve some problems
> > > > > - How to enable and disable plugins on a production system but
> > > > > I'm
> not
> > > > > quite sure how reliable that is.  Even eclipse asks you to
> > > > > restart
> eclipse
> > > > > after adding a plugin.
> > > > >
> > > >
> > > > I know the answer that Marcel would say about this point :D
> > >
> > > I just talked to someone who says it does work reliably in their
> environments.
> > >
> > > >
> > > >
> > > > >
> > > > > I think after looking at this, then I decided that
> > > > >
> > > > > - For modularity, nothing is better than compilation boundaries.
>  The
> > > > > problem with some of the plugins is that it depends on
> > > > > cloud-core
> and
> > > > > cloud-server.  It shouldn't .  All plugins must build to
> > > > > cloud-api
> only.
> > > > >  Since all interfaces of CloudStack is in cloud-api (if you
> > > > > think
> about
> > > > > that then cloud-api is basically the OSGi bundle), that's
> sufficient to
> > > > > separate the plugins.
> > > > > - For lifecycle of plugins, it probably requires that we switch
> > > > > to
> deploy
> > > > > in something like Karaf before we can achieve runtime lifecycle
> changes.
> > > > >  I'm not sure it's entire necessary and it doesn't take care of
> > > > > a
> plugin's
> > > > > database versioning problem.
> > > > > - To resolve the other problems, we basically need to break
> cloudstack
> > > > > into separate processes.  Hence I've proposed the idea of
> disaggregating
> > > > > cloudstack.
> > > > >
> > > >
> > > > Again, I am not an OSGi expert, but from what you say it is more
> > > > about compile and build time and making the separation of what one
> > > > module should depend on what very clear and documented which we
> > > > already do in my company as we also have a huge stack and also
> > > > looked into OSGi which again a
> great
> > > > tool but when we don't need everything it offers, we exactly
> > > > needed
> what
> > > > you explained and we manage that mostly through good usage of
> > > > Maven which I know is tricky
> > > >
> > > > About the runtime aspects and database versioning I am afraid I am
> > > > not aware about the internals and the exact requirements and hence
> > > > I can't
> > > give
> > > > any input
> > > >
> > > > On another side, as in either case we need to *disaggregate* ACS,
> > > > we
> can
> > > > make the disaggregated module OSGi ready at least as a 1st step
> towards
> > > > assessing whether OSGi is the way we should go or not, and when it
> > > > is
> > > more
> > > > clear then we can either say no it is not the option we need or we
> will
> > > > then be ready for the full move to OSGi
> > > >
> > > > Thoughts ?
> > >
> > > My current thoughts are the large disaggregated CloudStack Services
> > > are going to be RESTful services.  I don't see any benefits for them
> > > to be
> OSGi
> > > components for the reasons I've already listed.
> > >
> > > However, each service still may deploy plugins.  Those plugins can
> > > be converted to OSGi.  It shouldn't be too difficult to do it as well.
> >
> > And that is if we see there's a good match with OSGi.  I can see for
> example that if people want to add more automation, background monitoring
etc, deploying and undeploying those on a live system can be very helpful.
>  That maybe one good use of it.
> >
> > --Alex

RE: [DISCUSS] OSGi framework for plugins and more?

[Donal Lafferty <do...@citrix.com>]

1.  Is each service going to be a separate process?

2.  Will the component model follow the current kernel / plugin strategy?

DL

-----Original Message-----
From: Mohammad Nour El-Din [mailto:nour.mohammad@gmail.com] 
Sent: 15 November 2012 5:43 PM
To: cloudstack-dev@incubator.apache.org
Subject: RE: [DISCUSS] OSGi framework for plugins and more?

sounds like a good plan. allow me to brief what has been said:

1- disaggregatio of ACS into a set of services which are accessed restfully
2- the components of these services can be implemented as OSGi or at least OSGi ready

any missing points ? additional ones ? other insights ?

Sent from my Samsung Galaxy S3
Apologies for any typos
On Nov 15, 2012 6:20 PM, "Alex Huang" <Al...@citrix.com> wrote:
>
>
>
> > -----Original Message-----
> > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > Sent: Thursday, November 15, 2012 9:15 AM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: [DISCUSS] OSGi framework for plugins and more?
> >
> >
> >
> > > -----Original Message-----
> > > From: Mohammad Nour El-Din [mailto:nour.mohammad@gmail.com]
> > > Sent: Wednesday, November 14, 2012 7:11 AM
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: Re: [DISCUSS] OSGi framework for plugins and more?
> > >
> > > Hi Alex...
> > >
> > >    Thanks for opening the discussion in the direction of what we 
> > > need
to do
> > > and how to do it, I thought no one will respond to my request :D
> > >
> > > On Wed, Nov 14, 2012 at 3:45 PM, Alex Huang 
> > > <Al...@citrix.com>
> > > wrote:
> > >
> > > > Here's my two-bits on OSGi.  I actually started a thread like 
> > > > this sometime ago to which Mohammad reference.  I then did some 
> > > > research
> > > on what
> > > > OSGi can do.  The problems I looked at using OSGi for just isn't
solvable
> > > > by OSGi.
> > > >
> > > > For example,
> > > > - How to contain fault within an availability zone.
> > > > - How to do rolling upgrade and phase out the rolling upgrades 
> > > > over
a
> > span
> > > > of days to deal with the time that it might take.
> > > > - How to do database upgrades/downgrades for the plugins
> > > > - How to scale different components differently
> > > >
> > > > OSGi does solve some problems
> > > > - How to enable and disable plugins on a production system but 
> > > > I'm
not
> > > > quite sure how reliable that is.  Even eclipse asks you to 
> > > > restart
eclipse
> > > > after adding a plugin.
> > > >
> > >
> > > I know the answer that Marcel would say about this point :D
> >
> > I just talked to someone who says it does work reliably in their
environments.
> >
> > >
> > >
> > > >
> > > > I think after looking at this, then I decided that
> > > >
> > > > - For modularity, nothing is better than compilation boundaries.
 The
> > > > problem with some of the plugins is that it depends on 
> > > > cloud-core
and
> > > > cloud-server.  It shouldn't .  All plugins must build to 
> > > > cloud-api
only.
> > > >  Since all interfaces of CloudStack is in cloud-api (if you 
> > > > think
about
> > > > that then cloud-api is basically the OSGi bundle), that's
sufficient to
> > > > separate the plugins.
> > > > - For lifecycle of plugins, it probably requires that we switch 
> > > > to
deploy
> > > > in something like Karaf before we can achieve runtime lifecycle
changes.
> > > >  I'm not sure it's entire necessary and it doesn't take care of 
> > > > a
plugin's
> > > > database versioning problem.
> > > > - To resolve the other problems, we basically need to break
cloudstack
> > > > into separate processes.  Hence I've proposed the idea of
disaggregating
> > > > cloudstack.
> > > >
> > >
> > > Again, I am not an OSGi expert, but from what you say it is more 
> > > about compile and build time and making the separation of what one 
> > > module should depend on what very clear and documented which we 
> > > already do in my company as we also have a huge stack and also 
> > > looked into OSGi which again a
great
> > > tool but when we don't need everything it offers, we exactly 
> > > needed
what
> > > you explained and we manage that mostly through good usage of 
> > > Maven which I know is tricky
> > >
> > > About the runtime aspects and database versioning I am afraid I am 
> > > not aware about the internals and the exact requirements and hence 
> > > I can't
> > give
> > > any input
> > >
> > > On another side, as in either case we need to *disaggregate* ACS, 
> > > we
can
> > > make the disaggregated module OSGi ready at least as a 1st step
towards
> > > assessing whether OSGi is the way we should go or not, and when it 
> > > is
> > more
> > > clear then we can either say no it is not the option we need or we
will
> > > then be ready for the full move to OSGi
> > >
> > > Thoughts ?
> >
> > My current thoughts are the large disaggregated CloudStack Services 
> > are going to be RESTful services.  I don't see any benefits for them 
> > to be
OSGi
> > components for the reasons I've already listed.
> >
> > However, each service still may deploy plugins.  Those plugins can 
> > be converted to OSGi.  It shouldn't be too difficult to do it as well.
>
> And that is if we see there's a good match with OSGi.  I can see for
example that if people want to add more automation, background monitoring etc, deploying and undeploying those on a live system can be very helpful.
 That maybe one good use of it.
>
> --Alex

RE: [DISCUSS] OSGi framework for plugins and more?

[Mohammad Nour El-Din <no...@gmail.com>]

sounds like a good plan. allow me to brief what has been said:

1- disaggregatio of ACS into a set of services which are accessed restfully
2- the components of these services can be implemented as OSGi or at least
OSGi ready

any missing points ? additional ones ? other insights ?

Sent from my Samsung Galaxy S3
Apologies for any typos
On Nov 15, 2012 6:20 PM, "Alex Huang" <Al...@citrix.com> wrote:
>
>
>
> > -----Original Message-----
> > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > Sent: Thursday, November 15, 2012 9:15 AM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: [DISCUSS] OSGi framework for plugins and more?
> >
> >
> >
> > > -----Original Message-----
> > > From: Mohammad Nour El-Din [mailto:nour.mohammad@gmail.com]
> > > Sent: Wednesday, November 14, 2012 7:11 AM
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: Re: [DISCUSS] OSGi framework for plugins and more?
> > >
> > > Hi Alex...
> > >
> > >    Thanks for opening the discussion in the direction of what we need
to do
> > > and how to do it, I thought no one will respond to my request :D
> > >
> > > On Wed, Nov 14, 2012 at 3:45 PM, Alex Huang <Al...@citrix.com>
> > > wrote:
> > >
> > > > Here's my two-bits on OSGi.  I actually started a thread like this
> > > > sometime ago to which Mohammad reference.  I then did some research
> > > on what
> > > > OSGi can do.  The problems I looked at using OSGi for just isn't
solvable
> > > > by OSGi.
> > > >
> > > > For example,
> > > > - How to contain fault within an availability zone.
> > > > - How to do rolling upgrade and phase out the rolling upgrades over
a
> > span
> > > > of days to deal with the time that it might take.
> > > > - How to do database upgrades/downgrades for the plugins
> > > > - How to scale different components differently
> > > >
> > > > OSGi does solve some problems
> > > > - How to enable and disable plugins on a production system but I'm
not
> > > > quite sure how reliable that is.  Even eclipse asks you to restart
eclipse
> > > > after adding a plugin.
> > > >
> > >
> > > I know the answer that Marcel would say about this point :D
> >
> > I just talked to someone who says it does work reliably in their
environments.
> >
> > >
> > >
> > > >
> > > > I think after looking at this, then I decided that
> > > >
> > > > - For modularity, nothing is better than compilation boundaries.
 The
> > > > problem with some of the plugins is that it depends on cloud-core
and
> > > > cloud-server.  It shouldn't .  All plugins must build to cloud-api
only.
> > > >  Since all interfaces of CloudStack is in cloud-api (if you think
about
> > > > that then cloud-api is basically the OSGi bundle), that's
sufficient to
> > > > separate the plugins.
> > > > - For lifecycle of plugins, it probably requires that we switch to
deploy
> > > > in something like Karaf before we can achieve runtime lifecycle
changes.
> > > >  I'm not sure it's entire necessary and it doesn't take care of a
plugin's
> > > > database versioning problem.
> > > > - To resolve the other problems, we basically need to break
cloudstack
> > > > into separate processes.  Hence I've proposed the idea of
disaggregating
> > > > cloudstack.
> > > >
> > >
> > > Again, I am not an OSGi expert, but from what you say it is more about
> > > compile and build time and making the separation of what one module
> > > should
> > > depend on what very clear and documented which we already do in my
> > > company
> > > as we also have a huge stack and also looked into OSGi which again a
great
> > > tool but when we don't need everything it offers, we exactly needed
what
> > > you explained and we manage that mostly through good usage of Maven
> > > which I
> > > know is tricky
> > >
> > > About the runtime aspects and database versioning I am afraid I am not
> > > aware about the internals and the exact requirements and hence I can't
> > give
> > > any input
> > >
> > > On another side, as in either case we need to *disaggregate* ACS, we
can
> > > make the disaggregated module OSGi ready at least as a 1st step
towards
> > > assessing whether OSGi is the way we should go or not, and when it is
> > more
> > > clear then we can either say no it is not the option we need or we
will
> > > then be ready for the full move to OSGi
> > >
> > > Thoughts ?
> >
> > My current thoughts are the large disaggregated CloudStack Services are
> > going to be RESTful services.  I don't see any benefits for them to be
OSGi
> > components for the reasons I've already listed.
> >
> > However, each service still may deploy plugins.  Those plugins can be
> > converted to OSGi.  It shouldn't be too difficult to do it as well.
>
> And that is if we see there's a good match with OSGi.  I can see for
example that if people want to add more automation, background monitoring
etc, deploying and undeploying those on a live system can be very helpful.
 That maybe one good use of it.
>
> --Alex

RE: [DISCUSS] OSGi framework for plugins and more?

[Alex Huang <Al...@citrix.com>]

> -----Original Message-----
> From: Alex Huang [mailto:Alex.Huang@citrix.com]
> Sent: Thursday, November 15, 2012 9:15 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: [DISCUSS] OSGi framework for plugins and more?
> 
> 
> 
> > -----Original Message-----
> > From: Mohammad Nour El-Din [mailto:nour.mohammad@gmail.com]
> > Sent: Wednesday, November 14, 2012 7:11 AM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: Re: [DISCUSS] OSGi framework for plugins and more?
> >
> > Hi Alex...
> >
> >    Thanks for opening the discussion in the direction of what we need to do
> > and how to do it, I thought no one will respond to my request :D
> >
> > On Wed, Nov 14, 2012 at 3:45 PM, Alex Huang <Al...@citrix.com>
> > wrote:
> >
> > > Here's my two-bits on OSGi.  I actually started a thread like this
> > > sometime ago to which Mohammad reference.  I then did some research
> > on what
> > > OSGi can do.  The problems I looked at using OSGi for just isn't solvable
> > > by OSGi.
> > >
> > > For example,
> > > - How to contain fault within an availability zone.
> > > - How to do rolling upgrade and phase out the rolling upgrades over a
> span
> > > of days to deal with the time that it might take.
> > > - How to do database upgrades/downgrades for the plugins
> > > - How to scale different components differently
> > >
> > > OSGi does solve some problems
> > > - How to enable and disable plugins on a production system but I'm not
> > > quite sure how reliable that is.  Even eclipse asks you to restart eclipse
> > > after adding a plugin.
> > >
> >
> > I know the answer that Marcel would say about this point :D
> 
> I just talked to someone who says it does work reliably in their environments.
> 
> >
> >
> > >
> > > I think after looking at this, then I decided that
> > >
> > > - For modularity, nothing is better than compilation boundaries.  The
> > > problem with some of the plugins is that it depends on cloud-core and
> > > cloud-server.  It shouldn't .  All plugins must build to cloud-api only.
> > >  Since all interfaces of CloudStack is in cloud-api (if you think about
> > > that then cloud-api is basically the OSGi bundle), that's sufficient to
> > > separate the plugins.
> > > - For lifecycle of plugins, it probably requires that we switch to deploy
> > > in something like Karaf before we can achieve runtime lifecycle changes.
> > >  I'm not sure it's entire necessary and it doesn't take care of a plugin's
> > > database versioning problem.
> > > - To resolve the other problems, we basically need to break cloudstack
> > > into separate processes.  Hence I've proposed the idea of disaggregating
> > > cloudstack.
> > >
> >
> > Again, I am not an OSGi expert, but from what you say it is more about
> > compile and build time and making the separation of what one module
> > should
> > depend on what very clear and documented which we already do in my
> > company
> > as we also have a huge stack and also looked into OSGi which again a great
> > tool but when we don't need everything it offers, we exactly needed what
> > you explained and we manage that mostly through good usage of Maven
> > which I
> > know is tricky
> >
> > About the runtime aspects and database versioning I am afraid I am not
> > aware about the internals and the exact requirements and hence I can't
> give
> > any input
> >
> > On another side, as in either case we need to *disaggregate* ACS, we can
> > make the disaggregated module OSGi ready at least as a 1st step towards
> > assessing whether OSGi is the way we should go or not, and when it is
> more
> > clear then we can either say no it is not the option we need or we will
> > then be ready for the full move to OSGi
> >
> > Thoughts ?
> 
> My current thoughts are the large disaggregated CloudStack Services are
> going to be RESTful services.  I don't see any benefits for them to be OSGi
> components for the reasons I've already listed.
> 
> However, each service still may deploy plugins.  Those plugins can be
> converted to OSGi.  It shouldn't be too difficult to do it as well.

And that is if we see there's a good match with OSGi.  I can see for example that if people want to add more automation, background monitoring etc, deploying and undeploying those on a live system can be very helpful.  That maybe one good use of it.

--Alex

RE: [DISCUSS] OSGi framework for plugins and more?

[Alex Huang <Al...@citrix.com>]

> -----Original Message-----
> From: Mohammad Nour El-Din [mailto:nour.mohammad@gmail.com]
> Sent: Wednesday, November 14, 2012 7:11 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: [DISCUSS] OSGi framework for plugins and more?
> 
> Hi Alex...
> 
>    Thanks for opening the discussion in the direction of what we need to do
> and how to do it, I thought no one will respond to my request :D
> 
> On Wed, Nov 14, 2012 at 3:45 PM, Alex Huang <Al...@citrix.com>
> wrote:
> 
> > Here's my two-bits on OSGi.  I actually started a thread like this
> > sometime ago to which Mohammad reference.  I then did some research
> on what
> > OSGi can do.  The problems I looked at using OSGi for just isn't solvable
> > by OSGi.
> >
> > For example,
> > - How to contain fault within an availability zone.
> > - How to do rolling upgrade and phase out the rolling upgrades over a span
> > of days to deal with the time that it might take.
> > - How to do database upgrades/downgrades for the plugins
> > - How to scale different components differently
> >
> > OSGi does solve some problems
> > - How to enable and disable plugins on a production system but I'm not
> > quite sure how reliable that is.  Even eclipse asks you to restart eclipse
> > after adding a plugin.
> >
> 
> I know the answer that Marcel would say about this point :D

I just talked to someone who says it does work reliably in their environments. 

> 
> 
> >
> > I think after looking at this, then I decided that
> >
> > - For modularity, nothing is better than compilation boundaries.  The
> > problem with some of the plugins is that it depends on cloud-core and
> > cloud-server.  It shouldn't .  All plugins must build to cloud-api only.
> >  Since all interfaces of CloudStack is in cloud-api (if you think about
> > that then cloud-api is basically the OSGi bundle), that's sufficient to
> > separate the plugins.
> > - For lifecycle of plugins, it probably requires that we switch to deploy
> > in something like Karaf before we can achieve runtime lifecycle changes.
> >  I'm not sure it's entire necessary and it doesn't take care of a plugin's
> > database versioning problem.
> > - To resolve the other problems, we basically need to break cloudstack
> > into separate processes.  Hence I've proposed the idea of disaggregating
> > cloudstack.
> >
> 
> Again, I am not an OSGi expert, but from what you say it is more about
> compile and build time and making the separation of what one module
> should
> depend on what very clear and documented which we already do in my
> company
> as we also have a huge stack and also looked into OSGi which again a great
> tool but when we don't need everything it offers, we exactly needed what
> you explained and we manage that mostly through good usage of Maven
> which I
> know is tricky
> 
> About the runtime aspects and database versioning I am afraid I am not
> aware about the internals and the exact requirements and hence I can't give
> any input
> 
> On another side, as in either case we need to *disaggregate* ACS, we can
> make the disaggregated module OSGi ready at least as a 1st step towards
> assessing whether OSGi is the way we should go or not, and when it is more
> clear then we can either say no it is not the option we need or we will
> then be ready for the full move to OSGi
> 
> Thoughts ?

My current thoughts are the large disaggregated CloudStack Services are going to be RESTful services.  I don't see any benefits for them to be OSGi components for the reasons I've already listed.

However, each service still may deploy plugins.  Those plugins can be converted to OSGi.  It shouldn't be too difficult to do it as well.

--Alex

Re: [DISCUSS] OSGi framework for plugins and more?

[Mohammad Nour El-Din <no...@gmail.com>]

Hi Alex...

   Thanks for opening the discussion in the direction of what we need to do
and how to do it, I thought no one will respond to my request :D

On Wed, Nov 14, 2012 at 3:45 PM, Alex Huang <Al...@citrix.com> wrote:

> Here's my two-bits on OSGi.  I actually started a thread like this
> sometime ago to which Mohammad reference.  I then did some research on what
> OSGi can do.  The problems I looked at using OSGi for just isn't solvable
> by OSGi.
>
> For example,
> - How to contain fault within an availability zone.
> - How to do rolling upgrade and phase out the rolling upgrades over a span
> of days to deal with the time that it might take.
> - How to do database upgrades/downgrades for the plugins
> - How to scale different components differently
>
> OSGi does solve some problems
> - How to enable and disable plugins on a production system but I'm not
> quite sure how reliable that is.  Even eclipse asks you to restart eclipse
> after adding a plugin.
>

I know the answer that Marcel would say about this point :D


>
> I think after looking at this, then I decided that
>
> - For modularity, nothing is better than compilation boundaries.  The
> problem with some of the plugins is that it depends on cloud-core and
> cloud-server.  It shouldn't .  All plugins must build to cloud-api only.
>  Since all interfaces of CloudStack is in cloud-api (if you think about
> that then cloud-api is basically the OSGi bundle), that's sufficient to
> separate the plugins.
> - For lifecycle of plugins, it probably requires that we switch to deploy
> in something like Karaf before we can achieve runtime lifecycle changes.
>  I'm not sure it's entire necessary and it doesn't take care of a plugin's
> database versioning problem.
> - To resolve the other problems, we basically need to break cloudstack
> into separate processes.  Hence I've proposed the idea of disaggregating
> cloudstack.
>

Again, I am not an OSGi expert, but from what you say it is more about
compile and build time and making the separation of what one module should
depend on what very clear and documented which we already do in my company
as we also have a huge stack and also looked into OSGi which again a great
tool but when we don't need everything it offers, we exactly needed what
you explained and we manage that mostly through good usage of Maven which I
know is tricky

About the runtime aspects and database versioning I am afraid I am not
aware about the internals and the exact requirements and hence I can't give
any input

On another side, as in either case we need to *disaggregate* ACS, we can
make the disaggregated module OSGi ready at least as a 1st step towards
assessing whether OSGi is the way we should go or not, and when it is more
clear then we can either say no it is not the option we need or we will
then be ready for the full move to OSGi

Thoughts ?


>
> --Alex
>
> > -----Original Message-----
> > From: Charles Moulliard [mailto:ch007m@gmail.com]
> > Sent: Wednesday, November 14, 2012 1:41 AM
> > To: cloudstack-dev
> > Subject: Re: [DISCUSS] OSGi framework for plugins and more?
> >
> > Hi Kelven,
> >
> > As Spring/Spring DM can be used on Apache Karaf (multi-container OSGI
> > runtime), that should not be a problem to use what you have done on
> Karaf.
> > Maybe we can talk about that next on IRC channel (@ch007M)
> >
> > Regards,
> >
> >
> >
> >
> > On Tue, Nov 13, 2012 at 10:05 PM, Kelven Yang
> > <ke...@citrix.com>wrote:
> >
> > > The refactoring work that I did on Spring exploration is an incremental
> > > approach trying to improve and encourage modularity of CloudStack, I
> > hope
> > > it will help CloudStack in the future to add OSGi in the future.
> > >
> > > As we have a huge amount of legacy code base, it will be less risky to
> do
> > > incremental changes, and Spring effort here is the closest match of
> what
> > > we have in our existing code base. What we are trying to achieve is to
> > > first enable internal java modules to be more component-ized, our
> > > customized component framework actually provides 99% things we need,
> > > instead of us to fix the missing 1%, I feel that it might be worth to
> just
> > > switch to a more standard framework with a minimal effort. It is  that
> > > simple :-).  It is also easier to get things moving, in the end, if we
> > > have a well component-ized system internally, it will be easier to
> adopt
> > > any "the best" option like OSGi
> > >
> > > Kelven
> > >
> > >
> > > On 11/13/12 8:09 AM, "Mohammad Nour El-Din"
> > <no...@gmail.com>
> > > wrote:
> > >
> > > >Hi
> > > >
> > > >Had typos in this message the correct one is here:
> > > >
> > > >hi
> > > >
> > > >   we already had this discussion before earlier when ACS was just
> > donated
> > > >to ASF.
> > > >
> > > >I am not an OSGi expert at all :), but I would like to know what is
> the
> > > >problem that needs to be solved rather than discussing or jumping into
> > the
> > > >solution first
> > > >
> > > >OSGi has a lot of magnificent features idd but do we really need them
> all,
> > > >and also what does all these features add to us in the context of the
> > > >context of the cloud. to be honest I don't have a definite answer and
> > > >that's why I would suggest to list what we need to achieve regarding
> that
> > > >part and look into the alternatives and choose what is best for us,
> OSGi
> > > >is
> > > >one option for sure
> > > >
> > > >Feedback/input ?
> > > >On Tue, Nov 13, 2012 at 8:30 AM, Mohammad Nour El-Din <
> > > >nour.mohammad@gmail.com> wrote:
> > > >
> > > >> hi
> > > >>
> > > >>    we already had this discussion before earlier when ACS was just
> > > >>donated
> > > >> to ASF.
> > > >>
> > > >> I am not an OSGi expert at all :), but I would like to know what is
> the
> > > >> problem that needs to be solved rather than discussing the solution
> > > >>first
> > > >>
> > > >> OSGi has a lot of magnificent features idd but do we really need
> them
> > > >>all,
> > > >> and also what does all these features add to us in the context of
> the
> > > >> context of the cloud. to be honest I don have a definite answer and
> > > >>that's
> > > >> why I would suggest to list what we need to achieve regarding that
> part
> > > >>and
> > > >> look into the alternatives and choose what is best for us, OSGi is
> one
> > > >>of
> > > >> them for sure
> > > >>
> > > >> Feedback/input ?
> > > >>
> > > >> Sent from my Samsung Galaxy S3
> > > >> Apologies for any typos
> > > >>
> > > >> On Nov 13, 2012 7:56 AM, "Hugo Trippaers"
> > > >><HT...@schubergphilis.com>
> > > >> wrote:
> > > >> >
> > > >> >
> > > >> >
> > > >> > > -----Original Message-----
> > > >> > > From: Kelven Yang [mailto:kelven.yang@citrix.com]
> > > >> > > Sent: Monday, November 12, 2012 11:31 PM
> > > >> > > To: cloudstack-dev@incubator.apache.org
> > > >> > > Subject: Re: [DISCUSS] OSGi framework for plugins and more?
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On 11/12/12 1:34 PM, "Chip Childers" <chip.childers@sungard.com
> >
> > > >> wrote:
> > > >> > >
> > > >> > > >On Mon, Nov 12, 2012 at 4:12 AM, Hugo Trippaers
> > > >> > > ><HT...@schubergphilis.com> wrote:
> > > >> > > >> Hey  all,
> > > >> > > >>
> > > >> > > >> With the recent discussions on refactoring CloudStack and the
> > > >> working
> > > >> > > >>going into javelin I would like to discuss using OSGi. The
> > > >>background
> > > >> > > >>is that I have been struggling with ideas on how to setup a
> plugin
> > > >> > > >>system for CloudStack that would allow plugins to be separate
> > > >> entities
> > > >> > > >>which can be release independently from CloudStack core.
> Mainly
> > to
> > > >> > > >>deal with the current non-asf components but for future
> > expansion
> > > >>as
> > > >> > > well.
> > > >> > > >>
> > > >> > > >> While at ApacheConEU I had the chance to discuss these ideas
> > with
> > > >> one
> > > >> > > >>of our mentors after his talk about OSGi. I'm pretty charmed
> by
> > > >>OSGi
> > > >> > > >>and the options it provides. It's a well thought out system
> that
> > > >> allow
> > > >> > > >>true modularity and pluggability. With the amount of support
> in
> > > >>the
> > > >> > > >>java industry it's a standard that feels very mature and a
> safe
> > > >>bet,
> > > >> > > >>one I would prefer to any homegrown plugin system. It supports
> > > >> > > >>versioning of components, strict separation of classes between
> > > >> > > >>components and all kind of funky features like hot-plugging
> and
> > > >> > > >>hot-replace. Using OSGi would mean that people can supply
> > bundles
> > > >> with
> > > >> > > >>functionality which are maintained separately from the 'main
> > code'
> > > >> > > >>without having to worry about how to integrate it with the
> core.
> > > >>Just
> > > >> > > >>putting the module in the right directory should be enough to
> get
> > > >> > > CloudStack to see and use the bundle.
> > > >> > > >>Upgrades happen the same way, new version of an authenticator,
> > > >>just
> > > >> > > >>replace the bundle and let the framework replace it with
> having
> > to
> > > >> > > >>shutdown the server at all.
> > > >> > > >>
> > > >> > > >> As we are discussing making CloudStack more modular, I would
> > > >>like to
> > > >> > > >>propose to start using OSGi for this. It is a bit of a
> learning
> > > >>curve
> > > >> > > >>to start with, but one we can get help with from our mentors.
> I'm
> > > >> > > >>already working on setting up a proof of concept for a plugin
> > > >>system
> > > >> > > >>using OSGi together with a colleague to show how it works,
> code
> > is
> > > >> > > >>always better than words.
> > > >> > > >>
> > > >> > > >> So what are your thoughts?
> > > >> > > >>
> > > >> > > >> Cheers,
> > > >> > > >>
> > > >> > > >> Hugo
> > > >> > > >>
> > > >> > > >
> > > >> > > >I'm not familiar enough with OSGi to understand the tradeoffs
> of
> > > >>that
> > > >> > > >vs other frameworks, but I'd suggest that Kelvin weigh in here.
> > > >>The
> > > >> > > >work that he's doing on the Javelin branch is similar, and
> there
> > > >>might
> > > >> > > >be an argument for Spring instead.
> > > >> > > >
> > > >> > > >Kelvin, I know you just responded on the other thread about the
> > > >> > > >relative timing of a switch.  Want to weigh in on the OSGi
> > > >>approach's
> > > >> > > >technical merit vs. other options?
> > > >> > >
> > > >> > > It will be nice to see the OSGi technical merit vs. other
> option in
> > > >> details.
> > > >> > > Laying out these basic but fundamental frames may not benefit a
> lot
> > > >>in
> > > >> the
> > > >> > > short term, but we may get fully paid in long term. Spring can
> only
> > > >> give us
> > > >> > > solution on compile-time/load-time component integration, it
> > focuses
> > > >> > > more on internal component wiring, OSGi seems to focus more at
> > > >> runtime, I
> > > >> > > think these two may be complementary to each other.
> > > >> >
> > > >> > To a certain extent these technologies can be used together, but
> not
> > > >>in
> > > >> this way it seems. OSGi is a framework that focusses on separation
> of
> > > >>code
> > > >> in various modules in such a way that other modules can not see and
> > work
> > > >> with the classes in other modules excluding via well defined
> services.
> > > >>This
> > > >> is a fundamental choice that touches the core of how CloudStack
> would
> > be
> > > >> put together. Instead of a single codebase (or at runtime a single
> > > >> classloader) where modules would be loaded via (for example) the
> > spring
> > > >> framework based on an xml definition, the core would be an empty
> > > >>framework
> > > >> and modules work be plugged in and provide a certain service. This
> can
> > > >>be
> > > >> done at load time or runtime, that up to the implementers. For
> example
> > > >>say
> > > >> the core module would need a vm provisioned it would ask the service
> > > >> registry if there was a service able to provide this functionality,
> if
> > > >> there was that service would be asked to perform that task. Here is
> a
> > > >>short
> > > >> post that describes the context far better than I can:
> > > >> http://elron.us/?p=95 .The real benefits are in also in the
> modularity,
> > > >> because the framework is very strict on what bundles/interfaces are
> > > >>exposed
> > > >> and required, using the proper interfaces and limiting yourself to
> > > >>exposed
> > > >> interfaces is enforced by the framework.
> > > >> >
> > > >> > Hugo
> > > >> > >
> > > >> > >
> > > >> > > >
> > > >> > > >-chip
> > > >> >
> > > >>
> > > >>
> > > >
> > > >
> > > >--
> > > >Thanks
> > > >- Mohammad Nour
> > > >----
> > > >"Life is like riding a bicycle. To keep your balance you must keep
> moving"
> > > >- Albert Einstein
> > >
> > >
> >
> >
> > --
> > Charles Moulliard
> > Apache Committer / Sr. Enterprise Architect (RedHat)
> > Twitter : @cmoulliard | Blog : http://cmoulliard.blogspot.com
>



-- 
Thanks
- Mohammad Nour
----
"Life is like riding a bicycle. To keep your balance you must keep moving"
- Albert Einstein

RE: [DISCUSS] OSGi framework for plugins and more?

[Alex Huang <Al...@citrix.com>]

Here's my two-bits on OSGi.  I actually started a thread like this sometime ago to which Mohammad reference.  I then did some research on what OSGi can do.  The problems I looked at using OSGi for just isn't solvable by OSGi.

For example, 
- How to contain fault within an availability zone.
- How to do rolling upgrade and phase out the rolling upgrades over a span of days to deal with the time that it might take.
- How to do database upgrades/downgrades for the plugins
- How to scale different components differently

OSGi does solve some problems
- How to enable and disable plugins on a production system but I'm not quite sure how reliable that is.  Even eclipse asks you to restart eclipse after adding a plugin.

I think after looking at this, then I decided that 

- For modularity, nothing is better than compilation boundaries.  The problem with some of the plugins is that it depends on cloud-core and cloud-server.  It shouldn't .  All plugins must build to cloud-api only.  Since all interfaces of CloudStack is in cloud-api (if you think about that then cloud-api is basically the OSGi bundle), that's sufficient to separate the plugins.
- For lifecycle of plugins, it probably requires that we switch to deploy in something like Karaf before we can achieve runtime lifecycle changes.  I'm not sure it's entire necessary and it doesn't take care of a plugin's database versioning problem.
- To resolve the other problems, we basically need to break cloudstack into separate processes.  Hence I've proposed the idea of disaggregating cloudstack.

--Alex

> -----Original Message-----
> From: Charles Moulliard [mailto:ch007m@gmail.com]
> Sent: Wednesday, November 14, 2012 1:41 AM
> To: cloudstack-dev
> Subject: Re: [DISCUSS] OSGi framework for plugins and more?
> 
> Hi Kelven,
> 
> As Spring/Spring DM can be used on Apache Karaf (multi-container OSGI
> runtime), that should not be a problem to use what you have done on Karaf.
> Maybe we can talk about that next on IRC channel (@ch007M)
> 
> Regards,
> 
> 
> 
> 
> On Tue, Nov 13, 2012 at 10:05 PM, Kelven Yang
> <ke...@citrix.com>wrote:
> 
> > The refactoring work that I did on Spring exploration is an incremental
> > approach trying to improve and encourage modularity of CloudStack, I
> hope
> > it will help CloudStack in the future to add OSGi in the future.
> >
> > As we have a huge amount of legacy code base, it will be less risky to do
> > incremental changes, and Spring effort here is the closest match of what
> > we have in our existing code base. What we are trying to achieve is to
> > first enable internal java modules to be more component-ized, our
> > customized component framework actually provides 99% things we need,
> > instead of us to fix the missing 1%, I feel that it might be worth to just
> > switch to a more standard framework with a minimal effort. It is  that
> > simple :-).  It is also easier to get things moving, in the end, if we
> > have a well component-ized system internally, it will be easier to adopt
> > any "the best" option like OSGi
> >
> > Kelven
> >
> >
> > On 11/13/12 8:09 AM, "Mohammad Nour El-Din"
> <no...@gmail.com>
> > wrote:
> >
> > >Hi
> > >
> > >Had typos in this message the correct one is here:
> > >
> > >hi
> > >
> > >   we already had this discussion before earlier when ACS was just
> donated
> > >to ASF.
> > >
> > >I am not an OSGi expert at all :), but I would like to know what is the
> > >problem that needs to be solved rather than discussing or jumping into
> the
> > >solution first
> > >
> > >OSGi has a lot of magnificent features idd but do we really need them all,
> > >and also what does all these features add to us in the context of the
> > >context of the cloud. to be honest I don't have a definite answer and
> > >that's why I would suggest to list what we need to achieve regarding that
> > >part and look into the alternatives and choose what is best for us, OSGi
> > >is
> > >one option for sure
> > >
> > >Feedback/input ?
> > >On Tue, Nov 13, 2012 at 8:30 AM, Mohammad Nour El-Din <
> > >nour.mohammad@gmail.com> wrote:
> > >
> > >> hi
> > >>
> > >>    we already had this discussion before earlier when ACS was just
> > >>donated
> > >> to ASF.
> > >>
> > >> I am not an OSGi expert at all :), but I would like to know what is the
> > >> problem that needs to be solved rather than discussing the solution
> > >>first
> > >>
> > >> OSGi has a lot of magnificent features idd but do we really need them
> > >>all,
> > >> and also what does all these features add to us in the context of the
> > >> context of the cloud. to be honest I don have a definite answer and
> > >>that's
> > >> why I would suggest to list what we need to achieve regarding that part
> > >>and
> > >> look into the alternatives and choose what is best for us, OSGi is one
> > >>of
> > >> them for sure
> > >>
> > >> Feedback/input ?
> > >>
> > >> Sent from my Samsung Galaxy S3
> > >> Apologies for any typos
> > >>
> > >> On Nov 13, 2012 7:56 AM, "Hugo Trippaers"
> > >><HT...@schubergphilis.com>
> > >> wrote:
> > >> >
> > >> >
> > >> >
> > >> > > -----Original Message-----
> > >> > > From: Kelven Yang [mailto:kelven.yang@citrix.com]
> > >> > > Sent: Monday, November 12, 2012 11:31 PM
> > >> > > To: cloudstack-dev@incubator.apache.org
> > >> > > Subject: Re: [DISCUSS] OSGi framework for plugins and more?
> > >> > >
> > >> > >
> > >> > >
> > >> > > On 11/12/12 1:34 PM, "Chip Childers" <ch...@sungard.com>
> > >> wrote:
> > >> > >
> > >> > > >On Mon, Nov 12, 2012 at 4:12 AM, Hugo Trippaers
> > >> > > ><HT...@schubergphilis.com> wrote:
> > >> > > >> Hey  all,
> > >> > > >>
> > >> > > >> With the recent discussions on refactoring CloudStack and the
> > >> working
> > >> > > >>going into javelin I would like to discuss using OSGi. The
> > >>background
> > >> > > >>is that I have been struggling with ideas on how to setup a plugin
> > >> > > >>system for CloudStack that would allow plugins to be separate
> > >> entities
> > >> > > >>which can be release independently from CloudStack core. Mainly
> to
> > >> > > >>deal with the current non-asf components but for future
> expansion
> > >>as
> > >> > > well.
> > >> > > >>
> > >> > > >> While at ApacheConEU I had the chance to discuss these ideas
> with
> > >> one
> > >> > > >>of our mentors after his talk about OSGi. I'm pretty charmed by
> > >>OSGi
> > >> > > >>and the options it provides. It's a well thought out system that
> > >> allow
> > >> > > >>true modularity and pluggability. With the amount of support in
> > >>the
> > >> > > >>java industry it's a standard that feels very mature and a safe
> > >>bet,
> > >> > > >>one I would prefer to any homegrown plugin system. It supports
> > >> > > >>versioning of components, strict separation of classes between
> > >> > > >>components and all kind of funky features like hot-plugging and
> > >> > > >>hot-replace. Using OSGi would mean that people can supply
> bundles
> > >> with
> > >> > > >>functionality which are maintained separately from the 'main
> code'
> > >> > > >>without having to worry about how to integrate it with the core.
> > >>Just
> > >> > > >>putting the module in the right directory should be enough to get
> > >> > > CloudStack to see and use the bundle.
> > >> > > >>Upgrades happen the same way, new version of an authenticator,
> > >>just
> > >> > > >>replace the bundle and let the framework replace it with having
> to
> > >> > > >>shutdown the server at all.
> > >> > > >>
> > >> > > >> As we are discussing making CloudStack more modular, I would
> > >>like to
> > >> > > >>propose to start using OSGi for this. It is a bit of a learning
> > >>curve
> > >> > > >>to start with, but one we can get help with from our mentors. I'm
> > >> > > >>already working on setting up a proof of concept for a plugin
> > >>system
> > >> > > >>using OSGi together with a colleague to show how it works, code
> is
> > >> > > >>always better than words.
> > >> > > >>
> > >> > > >> So what are your thoughts?
> > >> > > >>
> > >> > > >> Cheers,
> > >> > > >>
> > >> > > >> Hugo
> > >> > > >>
> > >> > > >
> > >> > > >I'm not familiar enough with OSGi to understand the tradeoffs of
> > >>that
> > >> > > >vs other frameworks, but I'd suggest that Kelvin weigh in here.
> > >>The
> > >> > > >work that he's doing on the Javelin branch is similar, and there
> > >>might
> > >> > > >be an argument for Spring instead.
> > >> > > >
> > >> > > >Kelvin, I know you just responded on the other thread about the
> > >> > > >relative timing of a switch.  Want to weigh in on the OSGi
> > >>approach's
> > >> > > >technical merit vs. other options?
> > >> > >
> > >> > > It will be nice to see the OSGi technical merit vs. other option in
> > >> details.
> > >> > > Laying out these basic but fundamental frames may not benefit a lot
> > >>in
> > >> the
> > >> > > short term, but we may get fully paid in long term. Spring can only
> > >> give us
> > >> > > solution on compile-time/load-time component integration, it
> focuses
> > >> > > more on internal component wiring, OSGi seems to focus more at
> > >> runtime, I
> > >> > > think these two may be complementary to each other.
> > >> >
> > >> > To a certain extent these technologies can be used together, but not
> > >>in
> > >> this way it seems. OSGi is a framework that focusses on separation of
> > >>code
> > >> in various modules in such a way that other modules can not see and
> work
> > >> with the classes in other modules excluding via well defined services.
> > >>This
> > >> is a fundamental choice that touches the core of how CloudStack would
> be
> > >> put together. Instead of a single codebase (or at runtime a single
> > >> classloader) where modules would be loaded via (for example) the
> spring
> > >> framework based on an xml definition, the core would be an empty
> > >>framework
> > >> and modules work be plugged in and provide a certain service. This can
> > >>be
> > >> done at load time or runtime, that up to the implementers. For example
> > >>say
> > >> the core module would need a vm provisioned it would ask the service
> > >> registry if there was a service able to provide this functionality, if
> > >> there was that service would be asked to perform that task. Here is a
> > >>short
> > >> post that describes the context far better than I can:
> > >> http://elron.us/?p=95 .The real benefits are in also in the modularity,
> > >> because the framework is very strict on what bundles/interfaces are
> > >>exposed
> > >> and required, using the proper interfaces and limiting yourself to
> > >>exposed
> > >> interfaces is enforced by the framework.
> > >> >
> > >> > Hugo
> > >> > >
> > >> > >
> > >> > > >
> > >> > > >-chip
> > >> >
> > >>
> > >>
> > >
> > >
> > >--
> > >Thanks
> > >- Mohammad Nour
> > >----
> > >"Life is like riding a bicycle. To keep your balance you must keep moving"
> > >- Albert Einstein
> >
> >
> 
> 
> --
> Charles Moulliard
> Apache Committer / Sr. Enterprise Architect (RedHat)
> Twitter : @cmoulliard | Blog : http://cmoulliard.blogspot.com

Re: [DISCUSS] OSGi framework for plugins and more?

[Charles Moulliard <ch...@gmail.com>]

Hi Kelven,

As Spring/Spring DM can be used on Apache Karaf (multi-container OSGI
runtime), that should not be a problem to use what you have done on Karaf.
Maybe we can talk about that next on IRC channel (@ch007M)

Regards,




On Tue, Nov 13, 2012 at 10:05 PM, Kelven Yang <ke...@citrix.com>wrote:

> The refactoring work that I did on Spring exploration is an incremental
> approach trying to improve and encourage modularity of CloudStack, I hope
> it will help CloudStack in the future to add OSGi in the future.
>
> As we have a huge amount of legacy code base, it will be less risky to do
> incremental changes, and Spring effort here is the closest match of what
> we have in our existing code base. What we are trying to achieve is to
> first enable internal java modules to be more component-ized, our
> customized component framework actually provides 99% things we need,
> instead of us to fix the missing 1%, I feel that it might be worth to just
> switch to a more standard framework with a minimal effort. It is  that
> simple :-).  It is also easier to get things moving, in the end, if we
> have a well component-ized system internally, it will be easier to adopt
> any "the best" option like OSGi
>
> Kelven
>
>
> On 11/13/12 8:09 AM, "Mohammad Nour El-Din" <no...@gmail.com>
> wrote:
>
> >Hi
> >
> >Had typos in this message the correct one is here:
> >
> >hi
> >
> >   we already had this discussion before earlier when ACS was just donated
> >to ASF.
> >
> >I am not an OSGi expert at all :), but I would like to know what is the
> >problem that needs to be solved rather than discussing or jumping into the
> >solution first
> >
> >OSGi has a lot of magnificent features idd but do we really need them all,
> >and also what does all these features add to us in the context of the
> >context of the cloud. to be honest I don't have a definite answer and
> >that's why I would suggest to list what we need to achieve regarding that
> >part and look into the alternatives and choose what is best for us, OSGi
> >is
> >one option for sure
> >
> >Feedback/input ?
> >On Tue, Nov 13, 2012 at 8:30 AM, Mohammad Nour El-Din <
> >nour.mohammad@gmail.com> wrote:
> >
> >> hi
> >>
> >>    we already had this discussion before earlier when ACS was just
> >>donated
> >> to ASF.
> >>
> >> I am not an OSGi expert at all :), but I would like to know what is the
> >> problem that needs to be solved rather than discussing the solution
> >>first
> >>
> >> OSGi has a lot of magnificent features idd but do we really need them
> >>all,
> >> and also what does all these features add to us in the context of the
> >> context of the cloud. to be honest I don have a definite answer and
> >>that's
> >> why I would suggest to list what we need to achieve regarding that part
> >>and
> >> look into the alternatives and choose what is best for us, OSGi is one
> >>of
> >> them for sure
> >>
> >> Feedback/input ?
> >>
> >> Sent from my Samsung Galaxy S3
> >> Apologies for any typos
> >>
> >> On Nov 13, 2012 7:56 AM, "Hugo Trippaers"
> >><HT...@schubergphilis.com>
> >> wrote:
> >> >
> >> >
> >> >
> >> > > -----Original Message-----
> >> > > From: Kelven Yang [mailto:kelven.yang@citrix.com]
> >> > > Sent: Monday, November 12, 2012 11:31 PM
> >> > > To: cloudstack-dev@incubator.apache.org
> >> > > Subject: Re: [DISCUSS] OSGi framework for plugins and more?
> >> > >
> >> > >
> >> > >
> >> > > On 11/12/12 1:34 PM, "Chip Childers" <ch...@sungard.com>
> >> wrote:
> >> > >
> >> > > >On Mon, Nov 12, 2012 at 4:12 AM, Hugo Trippaers
> >> > > ><HT...@schubergphilis.com> wrote:
> >> > > >> Hey  all,
> >> > > >>
> >> > > >> With the recent discussions on refactoring CloudStack and the
> >> working
> >> > > >>going into javelin I would like to discuss using OSGi. The
> >>background
> >> > > >>is that I have been struggling with ideas on how to setup a plugin
> >> > > >>system for CloudStack that would allow plugins to be separate
> >> entities
> >> > > >>which can be release independently from CloudStack core. Mainly to
> >> > > >>deal with the current non-asf components but for future expansion
> >>as
> >> > > well.
> >> > > >>
> >> > > >> While at ApacheConEU I had the chance to discuss these ideas with
> >> one
> >> > > >>of our mentors after his talk about OSGi. I'm pretty charmed by
> >>OSGi
> >> > > >>and the options it provides. It's a well thought out system that
> >> allow
> >> > > >>true modularity and pluggability. With the amount of support in
> >>the
> >> > > >>java industry it's a standard that feels very mature and a safe
> >>bet,
> >> > > >>one I would prefer to any homegrown plugin system. It supports
> >> > > >>versioning of components, strict separation of classes between
> >> > > >>components and all kind of funky features like hot-plugging and
> >> > > >>hot-replace. Using OSGi would mean that people can supply bundles
> >> with
> >> > > >>functionality which are maintained separately from the 'main code'
> >> > > >>without having to worry about how to integrate it with the core.
> >>Just
> >> > > >>putting the module in the right directory should be enough to get
> >> > > CloudStack to see and use the bundle.
> >> > > >>Upgrades happen the same way, new version of an authenticator,
> >>just
> >> > > >>replace the bundle and let the framework replace it with having to
> >> > > >>shutdown the server at all.
> >> > > >>
> >> > > >> As we are discussing making CloudStack more modular, I would
> >>like to
> >> > > >>propose to start using OSGi for this. It is a bit of a learning
> >>curve
> >> > > >>to start with, but one we can get help with from our mentors. I'm
> >> > > >>already working on setting up a proof of concept for a plugin
> >>system
> >> > > >>using OSGi together with a colleague to show how it works, code is
> >> > > >>always better than words.
> >> > > >>
> >> > > >> So what are your thoughts?
> >> > > >>
> >> > > >> Cheers,
> >> > > >>
> >> > > >> Hugo
> >> > > >>
> >> > > >
> >> > > >I'm not familiar enough with OSGi to understand the tradeoffs of
> >>that
> >> > > >vs other frameworks, but I'd suggest that Kelvin weigh in here.
> >>The
> >> > > >work that he's doing on the Javelin branch is similar, and there
> >>might
> >> > > >be an argument for Spring instead.
> >> > > >
> >> > > >Kelvin, I know you just responded on the other thread about the
> >> > > >relative timing of a switch.  Want to weigh in on the OSGi
> >>approach's
> >> > > >technical merit vs. other options?
> >> > >
> >> > > It will be nice to see the OSGi technical merit vs. other option in
> >> details.
> >> > > Laying out these basic but fundamental frames may not benefit a lot
> >>in
> >> the
> >> > > short term, but we may get fully paid in long term. Spring can only
> >> give us
> >> > > solution on compile-time/load-time component integration, it focuses
> >> > > more on internal component wiring, OSGi seems to focus more at
> >> runtime, I
> >> > > think these two may be complementary to each other.
> >> >
> >> > To a certain extent these technologies can be used together, but not
> >>in
> >> this way it seems. OSGi is a framework that focusses on separation of
> >>code
> >> in various modules in such a way that other modules can not see and work
> >> with the classes in other modules excluding via well defined services.
> >>This
> >> is a fundamental choice that touches the core of how CloudStack would be
> >> put together. Instead of a single codebase (or at runtime a single
> >> classloader) where modules would be loaded via (for example) the spring
> >> framework based on an xml definition, the core would be an empty
> >>framework
> >> and modules work be plugged in and provide a certain service. This can
> >>be
> >> done at load time or runtime, that up to the implementers. For example
> >>say
> >> the core module would need a vm provisioned it would ask the service
> >> registry if there was a service able to provide this functionality, if
> >> there was that service would be asked to perform that task. Here is a
> >>short
> >> post that describes the context far better than I can:
> >> http://elron.us/?p=95 .The real benefits are in also in the modularity,
> >> because the framework is very strict on what bundles/interfaces are
> >>exposed
> >> and required, using the proper interfaces and limiting yourself to
> >>exposed
> >> interfaces is enforced by the framework.
> >> >
> >> > Hugo
> >> > >
> >> > >
> >> > > >
> >> > > >-chip
> >> >
> >>
> >>
> >
> >
> >--
> >Thanks
> >- Mohammad Nour
> >----
> >"Life is like riding a bicycle. To keep your balance you must keep moving"
> >- Albert Einstein
>
>


-- 
Charles Moulliard
Apache Committer / Sr. Enterprise Architect (RedHat)
Twitter : @cmoulliard | Blog : http://cmoulliard.blogspot.com

Re: [DISCUSS] OSGi framework for plugins and more?

[Kelven Yang <ke...@citrix.com>]

The refactoring work that I did on Spring exploration is an incremental
approach trying to improve and encourage modularity of CloudStack, I hope
it will help CloudStack in the future to add OSGi in the future.

As we have a huge amount of legacy code base, it will be less risky to do
incremental changes, and Spring effort here is the closest match of what
we have in our existing code base. What we are trying to achieve is to
first enable internal java modules to be more component-ized, our
customized component framework actually provides 99% things we need,
instead of us to fix the missing 1%, I feel that it might be worth to just
switch to a more standard framework with a minimal effort. It is  that
simple :-).  It is also easier to get things moving, in the end, if we
have a well component-ized system internally, it will be easier to adopt
any "the best" option like OSGi

Kelven


On 11/13/12 8:09 AM, "Mohammad Nour El-Din" <no...@gmail.com>
wrote:

>Hi
>
>Had typos in this message the correct one is here:
>
>hi
>
>   we already had this discussion before earlier when ACS was just donated
>to ASF.
>
>I am not an OSGi expert at all :), but I would like to know what is the
>problem that needs to be solved rather than discussing or jumping into the
>solution first
>
>OSGi has a lot of magnificent features idd but do we really need them all,
>and also what does all these features add to us in the context of the
>context of the cloud. to be honest I don't have a definite answer and
>that's why I would suggest to list what we need to achieve regarding that
>part and look into the alternatives and choose what is best for us, OSGi
>is
>one option for sure
>
>Feedback/input ?
>On Tue, Nov 13, 2012 at 8:30 AM, Mohammad Nour El-Din <
>nour.mohammad@gmail.com> wrote:
>
>> hi
>>
>>    we already had this discussion before earlier when ACS was just
>>donated
>> to ASF.
>>
>> I am not an OSGi expert at all :), but I would like to know what is the
>> problem that needs to be solved rather than discussing the solution
>>first
>>
>> OSGi has a lot of magnificent features idd but do we really need them
>>all,
>> and also what does all these features add to us in the context of the
>> context of the cloud. to be honest I don have a definite answer and
>>that's
>> why I would suggest to list what we need to achieve regarding that part
>>and
>> look into the alternatives and choose what is best for us, OSGi is one
>>of
>> them for sure
>>
>> Feedback/input ?
>>
>> Sent from my Samsung Galaxy S3
>> Apologies for any typos
>>
>> On Nov 13, 2012 7:56 AM, "Hugo Trippaers"
>><HT...@schubergphilis.com>
>> wrote:
>> >
>> >
>> >
>> > > -----Original Message-----
>> > > From: Kelven Yang [mailto:kelven.yang@citrix.com]
>> > > Sent: Monday, November 12, 2012 11:31 PM
>> > > To: cloudstack-dev@incubator.apache.org
>> > > Subject: Re: [DISCUSS] OSGi framework for plugins and more?
>> > >
>> > >
>> > >
>> > > On 11/12/12 1:34 PM, "Chip Childers" <ch...@sungard.com>
>> wrote:
>> > >
>> > > >On Mon, Nov 12, 2012 at 4:12 AM, Hugo Trippaers
>> > > ><HT...@schubergphilis.com> wrote:
>> > > >> Hey  all,
>> > > >>
>> > > >> With the recent discussions on refactoring CloudStack and the
>> working
>> > > >>going into javelin I would like to discuss using OSGi. The
>>background
>> > > >>is that I have been struggling with ideas on how to setup a plugin
>> > > >>system for CloudStack that would allow plugins to be separate
>> entities
>> > > >>which can be release independently from CloudStack core. Mainly to
>> > > >>deal with the current non-asf components but for future expansion
>>as
>> > > well.
>> > > >>
>> > > >> While at ApacheConEU I had the chance to discuss these ideas with
>> one
>> > > >>of our mentors after his talk about OSGi. I'm pretty charmed by
>>OSGi
>> > > >>and the options it provides. It's a well thought out system that
>> allow
>> > > >>true modularity and pluggability. With the amount of support in
>>the
>> > > >>java industry it's a standard that feels very mature and a safe
>>bet,
>> > > >>one I would prefer to any homegrown plugin system. It supports
>> > > >>versioning of components, strict separation of classes between
>> > > >>components and all kind of funky features like hot-plugging and
>> > > >>hot-replace. Using OSGi would mean that people can supply bundles
>> with
>> > > >>functionality which are maintained separately from the 'main code'
>> > > >>without having to worry about how to integrate it with the core.
>>Just
>> > > >>putting the module in the right directory should be enough to get
>> > > CloudStack to see and use the bundle.
>> > > >>Upgrades happen the same way, new version of an authenticator,
>>just
>> > > >>replace the bundle and let the framework replace it with having to
>> > > >>shutdown the server at all.
>> > > >>
>> > > >> As we are discussing making CloudStack more modular, I would
>>like to
>> > > >>propose to start using OSGi for this. It is a bit of a learning
>>curve
>> > > >>to start with, but one we can get help with from our mentors. I'm
>> > > >>already working on setting up a proof of concept for a plugin
>>system
>> > > >>using OSGi together with a colleague to show how it works, code is
>> > > >>always better than words.
>> > > >>
>> > > >> So what are your thoughts?
>> > > >>
>> > > >> Cheers,
>> > > >>
>> > > >> Hugo
>> > > >>
>> > > >
>> > > >I'm not familiar enough with OSGi to understand the tradeoffs of
>>that
>> > > >vs other frameworks, but I'd suggest that Kelvin weigh in here.
>>The
>> > > >work that he's doing on the Javelin branch is similar, and there
>>might
>> > > >be an argument for Spring instead.
>> > > >
>> > > >Kelvin, I know you just responded on the other thread about the
>> > > >relative timing of a switch.  Want to weigh in on the OSGi
>>approach's
>> > > >technical merit vs. other options?
>> > >
>> > > It will be nice to see the OSGi technical merit vs. other option in
>> details.
>> > > Laying out these basic but fundamental frames may not benefit a lot
>>in
>> the
>> > > short term, but we may get fully paid in long term. Spring can only
>> give us
>> > > solution on compile-time/load-time component integration, it focuses
>> > > more on internal component wiring, OSGi seems to focus more at
>> runtime, I
>> > > think these two may be complementary to each other.
>> >
>> > To a certain extent these technologies can be used together, but not
>>in
>> this way it seems. OSGi is a framework that focusses on separation of
>>code
>> in various modules in such a way that other modules can not see and work
>> with the classes in other modules excluding via well defined services.
>>This
>> is a fundamental choice that touches the core of how CloudStack would be
>> put together. Instead of a single codebase (or at runtime a single
>> classloader) where modules would be loaded via (for example) the spring
>> framework based on an xml definition, the core would be an empty
>>framework
>> and modules work be plugged in and provide a certain service. This can
>>be
>> done at load time or runtime, that up to the implementers. For example
>>say
>> the core module would need a vm provisioned it would ask the service
>> registry if there was a service able to provide this functionality, if
>> there was that service would be asked to perform that task. Here is a
>>short
>> post that describes the context far better than I can:
>> http://elron.us/?p=95 .The real benefits are in also in the modularity,
>> because the framework is very strict on what bundles/interfaces are
>>exposed
>> and required, using the proper interfaces and limiting yourself to
>>exposed
>> interfaces is enforced by the framework.
>> >
>> > Hugo
>> > >
>> > >
>> > > >
>> > > >-chip
>> >
>>
>>
>
>
>-- 
>Thanks
>- Mohammad Nour
>----
>"Life is like riding a bicycle. To keep your balance you must keep moving"
>- Albert Einstein

Re: [DISCUSS] OSGi framework for plugins and more?

[Mohammad Nour El-Din <no...@gmail.com>]

Hi

Had typos in this message the correct one is here:

hi

   we already had this discussion before earlier when ACS was just donated
to ASF.

I am not an OSGi expert at all :), but I would like to know what is the
problem that needs to be solved rather than discussing or jumping into the
solution first

OSGi has a lot of magnificent features idd but do we really need them all,
and also what does all these features add to us in the context of the
context of the cloud. to be honest I don't have a definite answer and
that's why I would suggest to list what we need to achieve regarding that
part and look into the alternatives and choose what is best for us, OSGi is
one option for sure

Feedback/input ?
On Tue, Nov 13, 2012 at 8:30 AM, Mohammad Nour El-Din <
nour.mohammad@gmail.com> wrote:

> hi
>
>    we already had this discussion before earlier when ACS was just donated
> to ASF.
>
> I am not an OSGi expert at all :), but I would like to know what is the
> problem that needs to be solved rather than discussing the solution first
>
> OSGi has a lot of magnificent features idd but do we really need them all,
> and also what does all these features add to us in the context of the
> context of the cloud. to be honest I don have a definite answer and that's
> why I would suggest to list what we need to achieve regarding that part and
> look into the alternatives and choose what is best for us, OSGi is one of
> them for sure
>
> Feedback/input ?
>
> Sent from my Samsung Galaxy S3
> Apologies for any typos
>
> On Nov 13, 2012 7:56 AM, "Hugo Trippaers" <HT...@schubergphilis.com>
> wrote:
> >
> >
> >
> > > -----Original Message-----
> > > From: Kelven Yang [mailto:kelven.yang@citrix.com]
> > > Sent: Monday, November 12, 2012 11:31 PM
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: Re: [DISCUSS] OSGi framework for plugins and more?
> > >
> > >
> > >
> > > On 11/12/12 1:34 PM, "Chip Childers" <ch...@sungard.com>
> wrote:
> > >
> > > >On Mon, Nov 12, 2012 at 4:12 AM, Hugo Trippaers
> > > ><HT...@schubergphilis.com> wrote:
> > > >> Hey  all,
> > > >>
> > > >> With the recent discussions on refactoring CloudStack and the
> working
> > > >>going into javelin I would like to discuss using OSGi. The background
> > > >>is that I have been struggling with ideas on how to setup a plugin
> > > >>system for CloudStack that would allow plugins to be separate
> entities
> > > >>which can be release independently from CloudStack core. Mainly to
> > > >>deal with the current non-asf components but for future expansion as
> > > well.
> > > >>
> > > >> While at ApacheConEU I had the chance to discuss these ideas with
> one
> > > >>of our mentors after his talk about OSGi. I'm pretty charmed by OSGi
> > > >>and the options it provides. It's a well thought out system that
> allow
> > > >>true modularity and pluggability. With the amount of support in the
> > > >>java industry it's a standard that feels very mature and a safe bet,
> > > >>one I would prefer to any homegrown plugin system. It supports
> > > >>versioning of components, strict separation of classes between
> > > >>components and all kind of funky features like hot-plugging and
> > > >>hot-replace. Using OSGi would mean that people can supply bundles
> with
> > > >>functionality which are maintained separately from the 'main code'
> > > >>without having to worry about how to integrate it with the core. Just
> > > >>putting the module in the right directory should be enough to get
> > > CloudStack to see and use the bundle.
> > > >>Upgrades happen the same way, new version of an authenticator, just
> > > >>replace the bundle and let the framework replace it with having to
> > > >>shutdown the server at all.
> > > >>
> > > >> As we are discussing making CloudStack more modular, I would like to
> > > >>propose to start using OSGi for this. It is a bit of a learning curve
> > > >>to start with, but one we can get help with from our mentors. I'm
> > > >>already working on setting up a proof of concept for a plugin system
> > > >>using OSGi together with a colleague to show how it works, code is
> > > >>always better than words.
> > > >>
> > > >> So what are your thoughts?
> > > >>
> > > >> Cheers,
> > > >>
> > > >> Hugo
> > > >>
> > > >
> > > >I'm not familiar enough with OSGi to understand the tradeoffs of that
> > > >vs other frameworks, but I'd suggest that Kelvin weigh in here.  The
> > > >work that he's doing on the Javelin branch is similar, and there might
> > > >be an argument for Spring instead.
> > > >
> > > >Kelvin, I know you just responded on the other thread about the
> > > >relative timing of a switch.  Want to weigh in on the OSGi approach's
> > > >technical merit vs. other options?
> > >
> > > It will be nice to see the OSGi technical merit vs. other option in
> details.
> > > Laying out these basic but fundamental frames may not benefit a lot in
> the
> > > short term, but we may get fully paid in long term. Spring can only
> give us
> > > solution on compile-time/load-time component integration, it focuses
> > > more on internal component wiring, OSGi seems to focus more at
> runtime, I
> > > think these two may be complementary to each other.
> >
> > To a certain extent these technologies can be used together, but not in
> this way it seems. OSGi is a framework that focusses on separation of code
> in various modules in such a way that other modules can not see and work
> with the classes in other modules excluding via well defined services. This
> is a fundamental choice that touches the core of how CloudStack would be
> put together. Instead of a single codebase (or at runtime a single
> classloader) where modules would be loaded via (for example) the spring
> framework based on an xml definition, the core would be an empty framework
> and modules work be plugged in and provide a certain service. This can be
> done at load time or runtime, that up to the implementers. For example say
> the core module would need a vm provisioned it would ask the service
> registry if there was a service able to provide this functionality, if
> there was that service would be asked to perform that task. Here is a short
> post that describes the context far better than I can:
> http://elron.us/?p=95 .The real benefits are in also in the modularity,
> because the framework is very strict on what bundles/interfaces are exposed
> and required, using the proper interfaces and limiting yourself to exposed
> interfaces is enforced by the framework.
> >
> > Hugo
> > >
> > >
> > > >
> > > >-chip
> >
>
>


-- 
Thanks
- Mohammad Nour
----
"Life is like riding a bicycle. To keep your balance you must keep moving"
- Albert Einstein

RE: [DISCUSS] OSGi framework for plugins and more?

[Mohammad Nour El-Din <no...@gmail.com>]

hi

   we already had this discussion before earlier when ACS was just donated
to ASF.

I am not an OSGi expert at all :), but I would like to know what is the
problem that needs to be solved rather than discussing the solution first

OSGi has a lot of magnificent features idd but do we really need them all,
and also what does all these features add to us in the context of the
context of the cloud. to be honest I don have a definite answer and that's
why I would suggest to list what we need to achieve regarding that part and
look into the alternatives and choose what is best for us, OSGi is one of
them for sure

Feedback/input ?

Sent from my Samsung Galaxy S3
Apologies for any typos
On Nov 13, 2012 7:56 AM, "Hugo Trippaers" <HT...@schubergphilis.com>
wrote:
>
>
>
> > -----Original Message-----
> > From: Kelven Yang [mailto:kelven.yang@citrix.com]
> > Sent: Monday, November 12, 2012 11:31 PM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: Re: [DISCUSS] OSGi framework for plugins and more?
> >
> >
> >
> > On 11/12/12 1:34 PM, "Chip Childers" <ch...@sungard.com> wrote:
> >
> > >On Mon, Nov 12, 2012 at 4:12 AM, Hugo Trippaers
> > ><HT...@schubergphilis.com> wrote:
> > >> Hey  all,
> > >>
> > >> With the recent discussions on refactoring CloudStack and the working
> > >>going into javelin I would like to discuss using OSGi. The background
> > >>is that I have been struggling with ideas on how to setup a plugin
> > >>system for CloudStack that would allow plugins to be separate entities
> > >>which can be release independently from CloudStack core. Mainly to
> > >>deal with the current non-asf components but for future expansion as
> > well.
> > >>
> > >> While at ApacheConEU I had the chance to discuss these ideas with one
> > >>of our mentors after his talk about OSGi. I'm pretty charmed by OSGi
> > >>and the options it provides. It's a well thought out system that allow
> > >>true modularity and pluggability. With the amount of support in the
> > >>java industry it's a standard that feels very mature and a safe bet,
> > >>one I would prefer to any homegrown plugin system. It supports
> > >>versioning of components, strict separation of classes between
> > >>components and all kind of funky features like hot-plugging and
> > >>hot-replace. Using OSGi would mean that people can supply bundles with
> > >>functionality which are maintained separately from the 'main code'
> > >>without having to worry about how to integrate it with the core. Just
> > >>putting the module in the right directory should be enough to get
> > CloudStack to see and use the bundle.
> > >>Upgrades happen the same way, new version of an authenticator, just
> > >>replace the bundle and let the framework replace it with having to
> > >>shutdown the server at all.
> > >>
> > >> As we are discussing making CloudStack more modular, I would like to
> > >>propose to start using OSGi for this. It is a bit of a learning curve
> > >>to start with, but one we can get help with from our mentors. I'm
> > >>already working on setting up a proof of concept for a plugin system
> > >>using OSGi together with a colleague to show how it works, code is
> > >>always better than words.
> > >>
> > >> So what are your thoughts?
> > >>
> > >> Cheers,
> > >>
> > >> Hugo
> > >>
> > >
> > >I'm not familiar enough with OSGi to understand the tradeoffs of that
> > >vs other frameworks, but I'd suggest that Kelvin weigh in here.  The
> > >work that he's doing on the Javelin branch is similar, and there might
> > >be an argument for Spring instead.
> > >
> > >Kelvin, I know you just responded on the other thread about the
> > >relative timing of a switch.  Want to weigh in on the OSGi approach's
> > >technical merit vs. other options?
> >
> > It will be nice to see the OSGi technical merit vs. other option in
details.
> > Laying out these basic but fundamental frames may not benefit a lot in
the
> > short term, but we may get fully paid in long term. Spring can only
give us
> > solution on compile-time/load-time component integration, it focuses
> > more on internal component wiring, OSGi seems to focus more at runtime,
I
> > think these two may be complementary to each other.
>
> To a certain extent these technologies can be used together, but not in
this way it seems. OSGi is a framework that focusses on separation of code
in various modules in such a way that other modules can not see and work
with the classes in other modules excluding via well defined services. This
is a fundamental choice that touches the core of how CloudStack would be
put together. Instead of a single codebase (or at runtime a single
classloader) where modules would be loaded via (for example) the spring
framework based on an xml definition, the core would be an empty framework
and modules work be plugged in and provide a certain service. This can be
done at load time or runtime, that up to the implementers. For example say
the core module would need a vm provisioned it would ask the service
registry if there was a service able to provide this functionality, if
there was that service would be asked to perform that task. Here is a short
post that describes the context far better than I can:
http://elron.us/?p=95.The real benefits are in also in the modularity,
because the framework is
very strict on what bundles/interfaces are exposed and required, using the
proper interfaces and limiting yourself to exposed interfaces is enforced
by the framework.
>
> Hugo
> >
> >
> > >
> > >-chip
>

RE: [DISCUSS] OSGi framework for plugins and more?

[Hugo Trippaers <HT...@schubergphilis.com>]

> -----Original Message-----
> From: Kelven Yang [mailto:kelven.yang@citrix.com]
> Sent: Monday, November 12, 2012 11:31 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: [DISCUSS] OSGi framework for plugins and more?
> 
> 
> 
> On 11/12/12 1:34 PM, "Chip Childers" <ch...@sungard.com> wrote:
> 
> >On Mon, Nov 12, 2012 at 4:12 AM, Hugo Trippaers
> ><HT...@schubergphilis.com> wrote:
> >> Hey  all,
> >>
> >> With the recent discussions on refactoring CloudStack and the working
> >>going into javelin I would like to discuss using OSGi. The background
> >>is that I have been struggling with ideas on how to setup a plugin
> >>system for CloudStack that would allow plugins to be separate entities
> >>which can be release independently from CloudStack core. Mainly to
> >>deal with the current non-asf components but for future expansion as
> well.
> >>
> >> While at ApacheConEU I had the chance to discuss these ideas with one
> >>of our mentors after his talk about OSGi. I'm pretty charmed by OSGi
> >>and the options it provides. It's a well thought out system that allow
> >>true modularity and pluggability. With the amount of support in the
> >>java industry it's a standard that feels very mature and a safe bet,
> >>one I would prefer to any homegrown plugin system. It supports
> >>versioning of components, strict separation of classes between
> >>components and all kind of funky features like hot-plugging and
> >>hot-replace. Using OSGi would mean that people can supply bundles with
> >>functionality which are maintained separately from the 'main code'
> >>without having to worry about how to integrate it with the core. Just
> >>putting the module in the right directory should be enough to get
> CloudStack to see and use the bundle.
> >>Upgrades happen the same way, new version of an authenticator, just
> >>replace the bundle and let the framework replace it with having to
> >>shutdown the server at all.
> >>
> >> As we are discussing making CloudStack more modular, I would like to
> >>propose to start using OSGi for this. It is a bit of a learning curve
> >>to start with, but one we can get help with from our mentors. I'm
> >>already working on setting up a proof of concept for a plugin system
> >>using OSGi together with a colleague to show how it works, code is
> >>always better than words.
> >>
> >> So what are your thoughts?
> >>
> >> Cheers,
> >>
> >> Hugo
> >>
> >
> >I'm not familiar enough with OSGi to understand the tradeoffs of that
> >vs other frameworks, but I'd suggest that Kelvin weigh in here.  The
> >work that he's doing on the Javelin branch is similar, and there might
> >be an argument for Spring instead.
> >
> >Kelvin, I know you just responded on the other thread about the
> >relative timing of a switch.  Want to weigh in on the OSGi approach's
> >technical merit vs. other options?
> 
> It will be nice to see the OSGi technical merit vs. other option in details.
> Laying out these basic but fundamental frames may not benefit a lot in the
> short term, but we may get fully paid in long term. Spring can only give us
> solution on compile-time/load-time component integration, it focuses
> more on internal component wiring, OSGi seems to focus more at runtime, I
> think these two may be complementary to each other.

To a certain extent these technologies can be used together, but not in this way it seems. OSGi is a framework that focusses on separation of code in various modules in such a way that other modules can not see and work with the classes in other modules excluding via well defined services. This is a fundamental choice that touches the core of how CloudStack would be put together. Instead of a single codebase (or at runtime a single classloader) where modules would be loaded via (for example) the spring framework based on an xml definition, the core would be an empty framework and modules work be plugged in and provide a certain service. This can be done at load time or runtime, that up to the implementers. For example say the core module would need a vm provisioned it would ask the service registry if there was a service able to provide this functionality, if there was that service would be asked to perform that task. Here is a short post that describes the context far better than I can: http://elron.us/?p=95 .The real benefits are in also in the modularity, because the framework is very strict on what bundles/interfaces are exposed and required, using the proper interfaces and limiting yourself to exposed interfaces is enforced by the framework. 

Hugo
> 
> 
> >
> >-chip

[DISCUSS] Refactoring ways for components to interact

[Kelven Yang <ke...@citrix.com>]

This is a follow up email on the topic I posted before, I've already put
in a skeleton implementation for things I mentioned in this discussion
email.  I'd like to update the community about one of the coding flavors
that are supported in this implementation at RPC layer. Please feel free
to comment.

If you are using RPC as a client only,  assuming you already have wired a
RPC provider, these are some examples to use it

1) Async RPC call 

StartVMCommand cmd = new StartVMcommand();
Š
rpcProvider.newCall( <your target component endpoint
address>).setCommand("StartVM").setComandArg(cmd).setTimeout(10000)
.addCallbackListener(new RpcCallbackListener<StartVMAnswer>() {

public void onSuccess(StartVMAnswer anser)  {
  Š.
}

public void onFailure(RpcException e) {
Š
} 

}).apply();

2) Sync RPC call

StartVMCommand cmd = new StartVMcommand();
Š
RpcClientCall call = rpcProvider.newCall( <your target component endpoint
address>).setCommand("StartVM").setComandArg(cmd).setTimeout(10000).apply()
;
Try {
StartVMAnswer answer = call.get();
} catch (RpcTimeoutException e) {
    Š
} catch( RpcIOException e) {
   Š
}  catch(RpcException e) {
   Š
}

If you are writing a service RPC component, have a class that participates
the RPC framework by extending ComponentEndpoint class, ComponentEndpoint
also allows you to participate event bus signaling with annotations

public MyServiceEndpoint extends ComponentEndpoint  {

@RpcServiceHandler(command="StartVM")
void HandleStartVM(final RpcServerCall call) {

Final StartVM cmd = call.getCommandArg();
_executor.execute(new Runnable() {
public void run() {
StartVMAnswer answer;
Š
call.completeCall(answer);
}
});
}

@EventHandler(topic="network.prepare")
void onPrepareNetwork(String sender, String topic, Object args) {
Š
}
}


There are other supported ways in the skeleton framework as well, for
example, one way RPC or event-dispatching based style,  but it seems so
far the above style is what  most people prefers.



Kelven



On 11/13/12 6:20 PM, "Kelven Yang" <ke...@citrix.com> wrote:

>I'm now working on some generic architecture-refactoring work, I'd like to
>share what I'm doing, your comments and feedbacks are very welcome
>
>Currently there are two major types of components that have
>inter-component interaction beyond Java method-call-bindings, one is
>Resource and the other one is what we called managers. Resources usually
>residents within their tenant environment(agent at client side or
>management server itself at server side), and managers run directly inside
>management server. Resources and managers communicate each other by
>exchanging Commands and Answers.
>
>For rest of other modules (for example, so called
>plugins/Gurus/Elements/Adapters), they usually bind to corresponding
>managers through normal Java method-calls.
>
>From high level perspective, the architecture looks relatively clean,
>however, as we add more and more features, things start to become murky,
>the major issue of this architectures is that we are lack of a clear
>middle layer for various components to build up interactions, so we end up
>with things like, a network component might need to wire agent manager in
>order to communicate with a particular agent. And agent manager itself, in
>addition to provide a communication layer, it has baked in a lot of
>CloudStack business logic to define what a agent should behave like (i.e.,
>StartupCommand/RoutingStartupCommand, etc).
>
>I'll start to work on following things to make CloudStack reshape into
>multi-layered architecture, focusing on providing a uniformed grand for
>components to interactive with each other and give all components
>first-class citizenship within CloudStack system
>
>1) a low-level messaging layer
>This layer provides three major functions
>	- endpoint address space that across management servers (clustering will
>be embedded solely within this layer and be transparent to high level
>components)
>	- generic message delivery, Yes, just plain arbitrary message delivery
>	- protocol multiplexing, allow applications on top of
>
>2) Event bus for component signaling
>	A multiplexed layer on top of messaging layer, it provides
>publisher/subscriber pattern within management server node, cross
>management server nodes to provide light-weight signaling mechanism
>between components
>
>3) RPC/Async RPC layer
>	- A multiplexed layer on top of messaging layer, although components are
>free to use any other RPC mechanism, however, within CloudStack internal,
>this generic RPC/Async RPC facility may probably be used the most.
>	- sequence control
>
>4) Event dispatching mechanism for Async orchestration
>
>Kelven
>

[DISCUSS] Refactoring ways for components to interact

[Kelven Yang <ke...@citrix.com>]

I'm now working on some generic architecture-refactoring work, I'd like to
share what I'm doing, your comments and feedbacks are very welcome

Currently there are two major types of components that have
inter-component interaction beyond Java method-call-bindings, one is
Resource and the other one is what we called managers. Resources usually
residents within their tenant environment(agent at client side or
management server itself at server side), and managers run directly inside
management server. Resources and managers communicate each other by
exchanging Commands and Answers.

For rest of other modules (for example, so called
plugins/Gurus/Elements/Adapters), they usually bind to corresponding
managers through normal Java method-calls.

>From high level perspective, the architecture looks relatively clean,
however, as we add more and more features, things start to become murky,
the major issue of this architectures is that we are lack of a clear
middle layer for various components to build up interactions, so we end up
with things like, a network component might need to wire agent manager in
order to communicate with a particular agent. And agent manager itself, in
addition to provide a communication layer, it has baked in a lot of
CloudStack business logic to define what a agent should behave like (i.e.,
StartupCommand/RoutingStartupCommand, etc).

I'll start to work on following things to make CloudStack reshape into
multi-layered architecture, focusing on providing a uniformed grand for
components to interactive with each other and give all components
first-class citizenship within CloudStack system

1) a low-level messaging layer
This layer provides three major functions
	- endpoint address space that across management servers (clustering will
be embedded solely within this layer and be transparent to high level
components)
	- generic message delivery, Yes, just plain arbitrary message delivery
	- protocol multiplexing, allow applications on top of

2) Event bus for component signaling
	A multiplexed layer on top of messaging layer, it provides
publisher/subscriber pattern within management server node, cross
management server nodes to provide light-weight signaling mechanism
between components 

3) RPC/Async RPC layer
	- A multiplexed layer on top of messaging layer, although components are
free to use any other RPC mechanism, however, within CloudStack internal,
this generic RPC/Async RPC facility may probably be used the most.
	- sequence control

4) Event dispatching mechanism for Async orchestration

Kelven

Re: [DISCUSS] OSGi framework for plugins and more?

[Kelven Yang <ke...@citrix.com>]

On 11/12/12 1:34 PM, "Chip Childers" <ch...@sungard.com> wrote:

>On Mon, Nov 12, 2012 at 4:12 AM, Hugo Trippaers
><HT...@schubergphilis.com> wrote:
>> Hey  all,
>>
>> With the recent discussions on refactoring CloudStack and the working
>>going into javelin I would like to discuss using OSGi. The background is
>>that I have been struggling with ideas on how to setup a plugin system
>>for CloudStack that would allow plugins to be separate entities which
>>can be release independently from CloudStack core. Mainly to deal with
>>the current non-asf components but for future expansion as well.
>>
>> While at ApacheConEU I had the chance to discuss these ideas with one
>>of our mentors after his talk about OSGi. I'm pretty charmed by OSGi and
>>the options it provides. It's a well thought out system that allow true
>>modularity and pluggability. With the amount of support in the java
>>industry it's a standard that feels very mature and a safe bet, one I
>>would prefer to any homegrown plugin system. It supports versioning of
>>components, strict separation of classes between components and all kind
>>of funky features like hot-plugging and hot-replace. Using OSGi would
>>mean that people can supply bundles with functionality which are
>>maintained separately from the 'main code' without having to worry about
>>how to integrate it with the core. Just putting the module in the right
>>directory should be enough to get CloudStack to see and use the bundle.
>>Upgrades happen the same way, new version of an authenticator, just
>>replace the bundle and let the framework replace it with having to
>>shutdown the server at all.
>>
>> As we are discussing making CloudStack more modular, I would like to
>>propose to start using OSGi for this. It is a bit of a learning curve to
>>start with, but one we can get help with from our mentors. I'm already
>>working on setting up a proof of concept for a plugin system using OSGi
>>together with a colleague to show how it works, code is always better
>>than words.
>>
>> So what are your thoughts?
>>
>> Cheers,
>>
>> Hugo
>>
>
>I'm not familiar enough with OSGi to understand the tradeoffs of that
>vs other frameworks, but I'd suggest that Kelvin weigh in here.  The
>work that he's doing on the Javelin branch is similar, and there might
>be an argument for Spring instead.
>
>Kelvin, I know you just responded on the other thread about the
>relative timing of a switch.  Want to weigh in on the OSGi approach's
>technical merit vs. other options?

It will be nice to see the OSGi technical merit vs. other option in
details. Laying out these basic but fundamental frames may not benefit a
lot in the short term, but we may get fully paid in long term. Spring can
only give us solution on compile-time/load-time component integration, it
focuses more on internal component wiring, OSGi seems to focus more at
runtime, I think these two may be complementary to each other.


>
>-chip

Re: [DISCUSS] OSGi framework for plugins and more?

[Chip Childers <ch...@sungard.com>]

On Mon, Nov 12, 2012 at 4:12 AM, Hugo Trippaers
<HT...@schubergphilis.com> wrote:
> Hey  all,
>
> With the recent discussions on refactoring CloudStack and the working going into javelin I would like to discuss using OSGi. The background is that I have been struggling with ideas on how to setup a plugin system for CloudStack that would allow plugins to be separate entities which can be release independently from CloudStack core. Mainly to deal with the current non-asf components but for future expansion as well.
>
> While at ApacheConEU I had the chance to discuss these ideas with one of our mentors after his talk about OSGi. I'm pretty charmed by OSGi and the options it provides. It's a well thought out system that allow true modularity and pluggability. With the amount of support in the java industry it's a standard that feels very mature and a safe bet, one I would prefer to any homegrown plugin system. It supports versioning of components, strict separation of classes between components and all kind of funky features like hot-plugging and hot-replace. Using OSGi would mean that people can supply bundles with functionality which are maintained separately from the 'main code' without having to worry about how to integrate it with the core. Just putting the module in the right directory should be enough to get CloudStack to see and use the bundle. Upgrades happen the same way, new version of an authenticator, just replace the bundle and let the framework replace it with having to shutdown the server at all.
>
> As we are discussing making CloudStack more modular, I would like to propose to start using OSGi for this. It is a bit of a learning curve to start with, but one we can get help with from our mentors. I'm already working on setting up a proof of concept for a plugin system using OSGi together with a colleague to show how it works, code is always better than words.
>
> So what are your thoughts?
>
> Cheers,
>
> Hugo
>

I'm not familiar enough with OSGi to understand the tradeoffs of that
vs other frameworks, but I'd suggest that Kelvin weigh in here.  The
work that he's doing on the Javelin branch is similar, and there might
be an argument for Spring instead.

Kelvin, I know you just responded on the other thread about the
relative timing of a switch.  Want to weigh in on the OSGi approach's
technical merit vs. other options?

-chip