Brooklyn extension mechanism - proposal

[Ciprian Ciubotariu <ch...@gmx.net>]

I'm sure this subject was already discussed on the mailing list and within the 
Brooklyn community, but I do have some questions and some thoughts on how to 
enhance the current state of affairs.


PROPOSAL

My proposal is, in short, to shift the architectural focus of Brooklyn towards 
a plugin-based implementation, in my view supported by OSGi.

In detail, I'd suggest deploying the brooklyn core, the REST server and the 
web console as separate bundles inside an OSGi container, allowing for 
locations, entities and blueprints to be plugged in as separate bundles.

With the above in mind, we can create an online catalog of OSGi bundles that 
can be added to a running brooklyn instance. Each bundle can be maintained 
separately, possibly by 3rd parties, and submitted to the central catalog for 
testing and validation before being published.


PREMISE

I based my proposal on the knowledge I could gather by browsing the 
documentation available on site, the examples supplied with the brooklyn 
project in the example folder and other projects at 
https://github.com/brooklyncentral.

These I believe to be the resources any new contributor would come in contact 
with.

I managed to coalesce that there are currently 3 possible ways of extension:

1. deriving a new project from brooklyn-downstream-parent (such as clocker, 
brooklyn-ambari, brooklyn-spark, and the examples shipped within the brooklyn 
source code). This way is implicitly endorsed by the brooklyn maven archetype.

2. create a "dropin" to be placed in $BROOKLYN_HOME/lib/dropins, such as 
brooklyn-location-azure-cli. Unfortunately the project seems to be stagnant, 
and there is no documentation about dropins.

3. fork the incubator-brooklyn project for personal use, perhaps contributing 
changes to the apache repository

Are there any other ways?


The first procedure newcomers run into is #1, which is quite disconcerting, 
because 

- it lacks the opportunity for composability (to deploy spark with clocker 
we'd need a new project, brooklyn-spark-clocker, or maybe brooklyn-clocker-
spark)
- it prevents 3rd parties from contributing their work back to the community

Mechanism #3 does not scale well with regard to the community, since users can 
only collaborate via the main brooklyn repository.

Mechanism #2 seems best from a technical point of view, but is not advertised 
and documented at all. Also, it suffers from dependency versioning problems 
already addressed by OSGi.


Implementing the main extensibility mechanism via OSGi would allow independent 
development of various entities and blueprints, therefore allowing the 
community to grow faster.

It also addresses the problem of upgrading entities separately within a 
running brooklyn instance, instead of the need to restart the entire  
application.

Each bundle (entity, location) can be subject to a set of automated tests, 
have reports automatically generated and presented in the online catalog.


Thank you,

Ciprian

Re: [PROPOSAL] Brooklyn extension mechanism

[Ciprian Ciubotariu <ch...@gmx.net>]

Karaf sounds good as a base directory - it can be moved from the sandbox later 
on.

What I regard as the base for making brooklyn more easily extensible is #2, 
while path #1 will follow closely.

I suggest we start working on the base of #2 after the rename is complete, 
within the sandbox of release 0.8.x.

On Wednesday 22 July 2015 21:07:50 Hadrian Zbarcea wrote:
> I think there are at least a couple of proposals here. It may be better
> to discuss them in separate threads. One comment I have, which was also
> captured in other threads is the current complexity of Brooklyn which is
> a bit intimidating to newcomers. Separating concerns into separate
> projects will reduce the complexity significantly, allowing one to focus
> on a particular area (such as an engineer developing an entity, or users
> developing blueprints).
> 
> 1. Separating the webui and catalogue as separate (sub)projects. I am
> all for that, it'd be probably better to discuss details in a separate
> thread. Maybe even move them to separate git repos down the road.
> 
> 2. Being able to deploy entities/blueprints from a catalogue would
> require some dynamic loading and that's where OSGi indeed shines.
> Brooklyn already has some support for OSGi, but it could benefit a lot
> more from it. I would recommend going beyond just the framework and use
> karaf that has a ton of useful features [1], like security, dynamic
> configuration, blueprints (the aries kind) [2], extensible shell
> commands [3] and so forth.
> 
> I have some experience with karaf and if this idea gets traction in the
> community I volunteer to come up with a skeleton for the required
> (sub)projects. We could either start in the sandbox, or a separate
> 'karaf' or 'osgi' directory.
> 
> Thoughts?
> Hadrian
> 
> 
> [1] https://karaf.apache.org/snippets/karaf-overview.html
> [2] http://aries.apache.org/modules/blueprint.html
> [3] https://karaf.apache.org/manual/latest/commands/commands.html
> 
> On 07/22/2015 08:14 AM, Ciprian Ciubotariu wrote:
> > I'm sure this subject was already discussed on the mailing list and within
> > the Brooklyn community, but I do have some questions and some thoughts on
> > how to enhance the current state of affairs.
> > 
> > 
> > PROPOSAL
> > 
> > My proposal is, in short, to shift the architectural focus of Brooklyn
> > towards a plugin-based implementation, in my view supported by OSGi.
> > 
> > In detail, I'd suggest deploying the brooklyn core, the REST server and
> > the
> > web console as separate bundles inside an OSGi container, allowing for
> > locations, entities and blueprints to be plugged in as separate bundles.
> > 
> > With the above in mind, we can create an online catalog of OSGi bundles
> > that can be added to a running brooklyn instance. Each bundle can be
> > maintained separately, possibly by 3rd parties, and submitted to the
> > central catalog for testing and validation before being published.
> > 
> > 
> > PREMISE
> > 
> > I based my proposal on the knowledge I could gather by browsing the
> > documentation available on site, the examples supplied with the brooklyn
> > project in the example folder and other projects at
> > https://github.com/brooklyncentral.
> > 
> > These I believe to be the resources any new contributor would come in
> > contact with.
> > 
> > I managed to coalesce that there are currently 3 possible ways of
> > extension:
> > 
> > 1. deriving a new project from brooklyn-downstream-parent (such as
> > clocker,
> > brooklyn-ambari, brooklyn-spark, and the examples shipped within the
> > brooklyn source code). This way is implicitly endorsed by the brooklyn
> > maven archetype.
> > 
> > 2. create a "dropin" to be placed in $BROOKLYN_HOME/lib/dropins, such as
> > brooklyn-location-azure-cli. Unfortunately the project seems to be
> > stagnant, and there is no documentation about dropins.
> > 
> > 3. fork the incubator-brooklyn project for personal use, perhaps
> > contributing changes to the apache repository
> > 
> > Are there any other ways?
> > 
> > 
> > The first procedure newcomers run into is #1, which is quite
> > disconcerting,
> > because
> > 
> > - it lacks the opportunity for composability (to deploy spark with clocker
> > we'd need a new project, brooklyn-spark-clocker, or maybe
> > brooklyn-clocker-
> > spark)
> > - it prevents 3rd parties from contributing their work back to the
> > community
> > 
> > Mechanism #3 does not scale well with regard to the community, since users
> > can only collaborate via the main brooklyn repository.
> > 
> > Mechanism #2 seems best from a technical point of view, but is not
> > advertised and documented at all. Also, it suffers from dependency
> > versioning problems already addressed by OSGi.
> > 
> > 
> > Implementing the main extensibility mechanism via OSGi would allow
> > independent development of various entities and blueprints, therefore
> > allowing the community to grow faster.
> > 
> > It also addresses the problem of upgrading entities separately within a
> > running brooklyn instance, instead of the need to restart the entire
> > application.
> > 
> > Each bundle (entity, location) can be subject to a set of automated tests,
> > have reports automatically generated and presented in the online catalog.
> > 
> > 
> > Thank you,
> > 
> > Ciprian

Re: [PROPOSAL] Brooklyn extension mechanism

[Hadrian Zbarcea <hz...@gmail.com>]

I think there are at least a couple of proposals here. It may be better 
to discuss them in separate threads. One comment I have, which was also 
captured in other threads is the current complexity of Brooklyn which is 
a bit intimidating to newcomers. Separating concerns into separate 
projects will reduce the complexity significantly, allowing one to focus 
on a particular area (such as an engineer developing an entity, or users 
developing blueprints).

1. Separating the webui and catalogue as separate (sub)projects. I am 
all for that, it'd be probably better to discuss details in a separate 
thread. Maybe even move them to separate git repos down the road.

2. Being able to deploy entities/blueprints from a catalogue would 
require some dynamic loading and that's where OSGi indeed shines. 
Brooklyn already has some support for OSGi, but it could benefit a lot 
more from it. I would recommend going beyond just the framework and use 
karaf that has a ton of useful features [1], like security, dynamic 
configuration, blueprints (the aries kind) [2], extensible shell 
commands [3] and so forth.

I have some experience with karaf and if this idea gets traction in the 
community I volunteer to come up with a skeleton for the required 
(sub)projects. We could either start in the sandbox, or a separate 
'karaf' or 'osgi' directory.

Thoughts?
Hadrian


[1] https://karaf.apache.org/snippets/karaf-overview.html
[2] http://aries.apache.org/modules/blueprint.html
[3] https://karaf.apache.org/manual/latest/commands/commands.html


On 07/22/2015 08:14 AM, Ciprian Ciubotariu wrote:
> I'm sure this subject was already discussed on the mailing list and within the
> Brooklyn community, but I do have some questions and some thoughts on how to
> enhance the current state of affairs.
>
>
> PROPOSAL
>
> My proposal is, in short, to shift the architectural focus of Brooklyn towards
> a plugin-based implementation, in my view supported by OSGi.
>
> In detail, I'd suggest deploying the brooklyn core, the REST server and the
> web console as separate bundles inside an OSGi container, allowing for
> locations, entities and blueprints to be plugged in as separate bundles.
>
> With the above in mind, we can create an online catalog of OSGi bundles that
> can be added to a running brooklyn instance. Each bundle can be maintained
> separately, possibly by 3rd parties, and submitted to the central catalog for
> testing and validation before being published.
>
>
> PREMISE
>
> I based my proposal on the knowledge I could gather by browsing the
> documentation available on site, the examples supplied with the brooklyn
> project in the example folder and other projects at
> https://github.com/brooklyncentral.
>
> These I believe to be the resources any new contributor would come in contact
> with.
>
> I managed to coalesce that there are currently 3 possible ways of extension:
>
> 1. deriving a new project from brooklyn-downstream-parent (such as clocker,
> brooklyn-ambari, brooklyn-spark, and the examples shipped within the brooklyn
> source code). This way is implicitly endorsed by the brooklyn maven archetype.
>
> 2. create a "dropin" to be placed in $BROOKLYN_HOME/lib/dropins, such as
> brooklyn-location-azure-cli. Unfortunately the project seems to be stagnant,
> and there is no documentation about dropins.
>
> 3. fork the incubator-brooklyn project for personal use, perhaps contributing
> changes to the apache repository
>
> Are there any other ways?
>
>
> The first procedure newcomers run into is #1, which is quite disconcerting,
> because
>
> - it lacks the opportunity for composability (to deploy spark with clocker
> we'd need a new project, brooklyn-spark-clocker, or maybe brooklyn-clocker-
> spark)
> - it prevents 3rd parties from contributing their work back to the community
>
> Mechanism #3 does not scale well with regard to the community, since users can
> only collaborate via the main brooklyn repository.
>
> Mechanism #2 seems best from a technical point of view, but is not advertised
> and documented at all. Also, it suffers from dependency versioning problems
> already addressed by OSGi.
>
>
> Implementing the main extensibility mechanism via OSGi would allow independent
> development of various entities and blueprints, therefore allowing the
> community to grow faster.
>
> It also addresses the problem of upgrading entities separately within a
> running brooklyn instance, instead of the need to restart the entire
> application.
>
> Each bundle (entity, location) can be subject to a set of automated tests,
> have reports automatically generated and presented in the online catalog.
>
>
> Thank you,
>
> Ciprian
>