Re: [PROPOSAL] Brooklyn extension mechanism

[Hadrian Zbarcea <hz...@gmail.com>]

I think there are at least a couple of proposals here. It may be better 
to discuss them in separate threads. One comment I have, which was also 
captured in other threads is the current complexity of Brooklyn which is 
a bit intimidating to newcomers. Separating concerns into separate 
projects will reduce the complexity significantly, allowing one to focus 
on a particular area (such as an engineer developing an entity, or users 
developing blueprints).

1. Separating the webui and catalogue as separate (sub)projects. I am 
all for that, it'd be probably better to discuss details in a separate 
thread. Maybe even move them to separate git repos down the road.

2. Being able to deploy entities/blueprints from a catalogue would 
require some dynamic loading and that's where OSGi indeed shines. 
Brooklyn already has some support for OSGi, but it could benefit a lot 
more from it. I would recommend going beyond just the framework and use 
karaf that has a ton of useful features [1], like security, dynamic 
configuration, blueprints (the aries kind) [2], extensible shell 
commands [3] and so forth.

I have some experience with karaf and if this idea gets traction in the 
community I volunteer to come up with a skeleton for the required 
(sub)projects. We could either start in the sandbox, or a separate 
'karaf' or 'osgi' directory.

Thoughts?
Hadrian


[1] https://karaf.apache.org/snippets/karaf-overview.html
[2] http://aries.apache.org/modules/blueprint.html
[3] https://karaf.apache.org/manual/latest/commands/commands.html


On 07/22/2015 08:14 AM, Ciprian Ciubotariu wrote:
> I'm sure this subject was already discussed on the mailing list and within the
> Brooklyn community, but I do have some questions and some thoughts on how to
> enhance the current state of affairs.
>
>
> PROPOSAL
>
> My proposal is, in short, to shift the architectural focus of Brooklyn towards
> a plugin-based implementation, in my view supported by OSGi.
>
> In detail, I'd suggest deploying the brooklyn core, the REST server and the
> web console as separate bundles inside an OSGi container, allowing for
> locations, entities and blueprints to be plugged in as separate bundles.
>
> With the above in mind, we can create an online catalog of OSGi bundles that
> can be added to a running brooklyn instance. Each bundle can be maintained
> separately, possibly by 3rd parties, and submitted to the central catalog for
> testing and validation before being published.
>
>
> PREMISE
>
> I based my proposal on the knowledge I could gather by browsing the
> documentation available on site, the examples supplied with the brooklyn
> project in the example folder and other projects at
> https://github.com/brooklyncentral.
>
> These I believe to be the resources any new contributor would come in contact
> with.
>
> I managed to coalesce that there are currently 3 possible ways of extension:
>
> 1. deriving a new project from brooklyn-downstream-parent (such as clocker,
> brooklyn-ambari, brooklyn-spark, and the examples shipped within the brooklyn
> source code). This way is implicitly endorsed by the brooklyn maven archetype.
>
> 2. create a "dropin" to be placed in $BROOKLYN_HOME/lib/dropins, such as
> brooklyn-location-azure-cli. Unfortunately the project seems to be stagnant,
> and there is no documentation about dropins.
>
> 3. fork the incubator-brooklyn project for personal use, perhaps contributing
> changes to the apache repository
>
> Are there any other ways?
>
>
> The first procedure newcomers run into is #1, which is quite disconcerting,
> because
>
> - it lacks the opportunity for composability (to deploy spark with clocker
> we'd need a new project, brooklyn-spark-clocker, or maybe brooklyn-clocker-
> spark)
> - it prevents 3rd parties from contributing their work back to the community
>
> Mechanism #3 does not scale well with regard to the community, since users can
> only collaborate via the main brooklyn repository.
>
> Mechanism #2 seems best from a technical point of view, but is not advertised
> and documented at all. Also, it suffers from dependency versioning problems
> already addressed by OSGi.
>
>
> Implementing the main extensibility mechanism via OSGi would allow independent
> development of various entities and blueprints, therefore allowing the
> community to grow faster.
>
> It also addresses the problem of upgrading entities separately within a
> running brooklyn instance, instead of the need to restart the entire
> application.
>
> Each bundle (entity, location) can be subject to a set of automated tests,
> have reports automatically generated and presented in the online catalog.
>
>
> Thank you,
>
> Ciprian
>

Re: [PROPOSAL] Brooklyn extension mechanism

[Ciprian Ciubotariu <ch...@gmx.net>]

Karaf sounds good as a base directory - it can be moved from the sandbox later 
on.

What I regard as the base for making brooklyn more easily extensible is #2, 
while path #1 will follow closely.

I suggest we start working on the base of #2 after the rename is complete, 
within the sandbox of release 0.8.x.

On Wednesday 22 July 2015 21:07:50 Hadrian Zbarcea wrote:
> I think there are at least a couple of proposals here. It may be better
> to discuss them in separate threads. One comment I have, which was also
> captured in other threads is the current complexity of Brooklyn which is
> a bit intimidating to newcomers. Separating concerns into separate
> projects will reduce the complexity significantly, allowing one to focus
> on a particular area (such as an engineer developing an entity, or users
> developing blueprints).
> 
> 1. Separating the webui and catalogue as separate (sub)projects. I am
> all for that, it'd be probably better to discuss details in a separate
> thread. Maybe even move them to separate git repos down the road.
> 
> 2. Being able to deploy entities/blueprints from a catalogue would
> require some dynamic loading and that's where OSGi indeed shines.
> Brooklyn already has some support for OSGi, but it could benefit a lot
> more from it. I would recommend going beyond just the framework and use
> karaf that has a ton of useful features [1], like security, dynamic
> configuration, blueprints (the aries kind) [2], extensible shell
> commands [3] and so forth.
> 
> I have some experience with karaf and if this idea gets traction in the
> community I volunteer to come up with a skeleton for the required
> (sub)projects. We could either start in the sandbox, or a separate
> 'karaf' or 'osgi' directory.
> 
> Thoughts?
> Hadrian
> 
> 
> [1] https://karaf.apache.org/snippets/karaf-overview.html
> [2] http://aries.apache.org/modules/blueprint.html
> [3] https://karaf.apache.org/manual/latest/commands/commands.html
> 
> On 07/22/2015 08:14 AM, Ciprian Ciubotariu wrote:
> > I'm sure this subject was already discussed on the mailing list and within
> > the Brooklyn community, but I do have some questions and some thoughts on
> > how to enhance the current state of affairs.
> > 
> > 
> > PROPOSAL
> > 
> > My proposal is, in short, to shift the architectural focus of Brooklyn
> > towards a plugin-based implementation, in my view supported by OSGi.
> > 
> > In detail, I'd suggest deploying the brooklyn core, the REST server and
> > the
> > web console as separate bundles inside an OSGi container, allowing for
> > locations, entities and blueprints to be plugged in as separate bundles.
> > 
> > With the above in mind, we can create an online catalog of OSGi bundles
> > that can be added to a running brooklyn instance. Each bundle can be
> > maintained separately, possibly by 3rd parties, and submitted to the
> > central catalog for testing and validation before being published.
> > 
> > 
> > PREMISE
> > 
> > I based my proposal on the knowledge I could gather by browsing the
> > documentation available on site, the examples supplied with the brooklyn
> > project in the example folder and other projects at
> > https://github.com/brooklyncentral.
> > 
> > These I believe to be the resources any new contributor would come in
> > contact with.
> > 
> > I managed to coalesce that there are currently 3 possible ways of
> > extension:
> > 
> > 1. deriving a new project from brooklyn-downstream-parent (such as
> > clocker,
> > brooklyn-ambari, brooklyn-spark, and the examples shipped within the
> > brooklyn source code). This way is implicitly endorsed by the brooklyn
> > maven archetype.
> > 
> > 2. create a "dropin" to be placed in $BROOKLYN_HOME/lib/dropins, such as
> > brooklyn-location-azure-cli. Unfortunately the project seems to be
> > stagnant, and there is no documentation about dropins.
> > 
> > 3. fork the incubator-brooklyn project for personal use, perhaps
> > contributing changes to the apache repository
> > 
> > Are there any other ways?
> > 
> > 
> > The first procedure newcomers run into is #1, which is quite
> > disconcerting,
> > because
> > 
> > - it lacks the opportunity for composability (to deploy spark with clocker
> > we'd need a new project, brooklyn-spark-clocker, or maybe
> > brooklyn-clocker-
> > spark)
> > - it prevents 3rd parties from contributing their work back to the
> > community
> > 
> > Mechanism #3 does not scale well with regard to the community, since users
> > can only collaborate via the main brooklyn repository.
> > 
> > Mechanism #2 seems best from a technical point of view, but is not
> > advertised and documented at all. Also, it suffers from dependency
> > versioning problems already addressed by OSGi.
> > 
> > 
> > Implementing the main extensibility mechanism via OSGi would allow
> > independent development of various entities and blueprints, therefore
> > allowing the community to grow faster.
> > 
> > It also addresses the problem of upgrading entities separately within a
> > running brooklyn instance, instead of the need to restart the entire
> > application.
> > 
> > Each bundle (entity, location) can be subject to a set of automated tests,
> > have reports automatically generated and presented in the online catalog.
> > 
> > 
> > Thank you,
> > 
> > Ciprian