You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by Alex Heneveld <al...@cloudsoftcorp.com> on 2017/06/19 16:39:51 UTC
Bundles in Brooklyn
Hi All-
With bundles now being neatly supported for add and remove, I think it
makes sense to use make the "bundle" a first-class concept in managing a
brooklyn catalog (type registry).
The basic idea is that users will work with ZIPs (or directories if
using the br CLI), where a "catalog.bom" is in the root; this is a
bundle, and operations like add/remove/update apply to the bundle.
Increasingly people are doing this for additions, so it seems pretty
obvious that delete should follow suit, and something like a "/bundles"
REST endpoint is appropriate.
You can still upload just a catalog.bom file; in this proposal it will
be wrapped in a bundle (internally, hidden from user). A bundle name
and version will have to be inferred/guessed for backwards
compatibility, and probably we should deprecate supplying a BOM which
doesn't define a bundle name and version.
If a ZIP/folder is supplied, it doesn't have to contain any OSGi bundle
definition; that call all be inferred. If a MANIFEST.MF is supplied,
any bundle name/version given there must of course match that in the
catalog.bom file. If dependent bundles are declared, they are obeyed at
runtime and can be used to find things (pretty much identically to how
we treat "libraries" in the catalog.bom.)
There are two subtle issues that arise:
* Versions - It would normally make sense for everything in a bundle to
have the same version number, viz. the version number of the bundle. We
don't need to enforce this, but I'm thinking we should deprecate and
WARN if this is the case because I don't think there's any compelling
reason to do otherwise. There is a question mark over what this version
should look like, because bundles must follow OSGi versioning and that
is different to strict semver and to the very casual "almost anything
you want" versioning style of maven and permitted by Brooklyn so far. I
wlll send a follow-up thread on this topic.
* Search Paths - Bundles fix an issue where types have implicit version
dependencies. Consider an "acme-cluster" of "acme-node" entities,
defined in a bundle. If I release a 1.0 version of this, and then a 2.0
version, I'd probably expect the 1.0 acme-cluster to load 1.0 acme-node
entities, and same for 2.0. But currently any reference to "acme-node"
takes the latest version, even if there is a different version in the
same bundle. Shifting to bundles, I think we should change this, so
that versions of items in the same bundle (or on the OSGi or library
search path for that bundle) are preferred over more recent versions in
the repository. (And since we will follow OSGi loading, we can also, if
we need to, use OSGi config to say that "acme" bundle requires a given
version _range_ for some other bundle, to have more control over
dependency versions when required.)
For most users and blueprints this will cause no changes. However there
are some changes to note:
* If people have been sloppy with versions or metadata, in most cases
they will get a warning
* In some cases if uploading a ZIP, the catalog.bom may need to be
edited (the upload will give a clear error, and it should not expect rebind)
* When a type is referenced with no version, the loading semantics may
change if that type is in the same bundle (this is probably what the
user expects)
Does anyone oppose any of the above, or have further questions?
Best
Alex
[1] https://github.com/apache/brooklyn-server/pull/672
Re: Bundles in Brooklyn
Posted by Alex Heneveld <al...@cloudsoftcorp.com>.
Aled-
Hi -- inline
On 22/06/2017 18:51, Aled Sage wrote:
> +1 to this approach, but I suspect there's a lot more to discuss/plan.
Great list of questions. We don't need these solved now for the
immediate proposal but I think we are reaching a consensus even on
tricky longer-term issues which is a good sanity check for the proposal
and hopefully means more people can work to take us in this direction!
>
> _*Leaking OSGi*_
> I agree with Geoff's worry of us "leaking OSGi". However, we should
> just "leak" the semantics of bundle dependencies and versioning, which
> need to be explained to the user. I think we can avoid exposing the
> OSGi syntax anywhere (except for power-users who want it).
Agree. Main question mark is whether we want to reuse (and expose) OSGi
for complex dependency situations e.g. ranges. But we don't need to
decide this now.
> _*`brooklyn.libraries` versus `package-imports analogy*_
> OSGi recommends the use of package-imports, rather than explicitly
> depending on specific bundles.
>
> The analogy for us would be to depend on other catalog item
> name:versionRange (e.g. in a `brooklyn.imports` section, or whatever).
> We'd then figure out which bundle to get those from using the same
> rules as OSGi package-imports.
>
> Is it worth exploring further that option, rather than going down the
> `brooklyn.libraries` route?
I think packages are wrong here. In Java they're a natural namespace
item which existed before bundles (and often repeated in bundles). In
our world we don't use packages for most type names and feedback has
been people don't want them. The bundles are the namespace -- so
"require-bundle" feels like the right OSGi concept.
(I confess for a while I thought "import-package" was an unnecessary
extra layer of abstraction in osgi -- always found the idea cool and
technical implementation is awesome but the vast majority of packages
come from one bundle anyway. The exceptions I've seen are things like
javax.* which really should be in their own bundle. Basing on packages
makes the wiring a lot more complicated to understand and debug.
However my opinion changed when I realized how clever OSGi is in looking
at what dependent packages are _exposed_ by a set of packages: e.g. if
B:b.internal.Clazz.method() returns type C:c.Class that makes C a
dependency of anything using B, whereas if something just used B:b and
not B:b.internal then C is not an exported dependency. Of course with
Brooklyn types we don't have or need that level of precision: so I
remain at the conclusion that "import-package" would _not_ be worth its
weight in our world.)
> _*References Between Bundles*_
> I don't think long-term the `brooklyn.libraries` should have URLs of
> other bundles. It could give the name:versionRange of other bundles
> (which assumes they have somehow already been added to Brooklyn).
>
> (Reasoning is that other smart folk don't embed the URLs, presumably
> because they want people to chose which repo things will come from -
> e.g. enterprise private repo, versus maven central, etc).
This argues for being able to configure Brooklyn (Karaf) to be aware of
repositories from which dependent bundles can be pulled. I think that
gets my vote longer term, instead of URLs. This could be made to work
with either `brooklyn.libraries` or `require-bundle`. But again we
don't need to decide now.
> _*Deploying Multiple Bundles*_
> I lean towards something analogous to the kar file, but maybe lighter
> weight rather than introducing anything like "features".
>
> Really we just need a list of bundle URLs. If someone then deletes one
> of those bundles, which breaks the other bundles deployed along with
> it, then tough!
>
Using repos feels like the best answer to me.
> _*Blueprint Validation*_
> Longer term, it would be nice if we further improved our blueprint
> validation to ensure all references in a catalog item being added are
> actually resolvable.
>
> Adding to Brooklyn the ability to add a list-of-bundles will no doubt
> make that harder!
>
+1
> _*Incremental Steps?*_
> This might well prove to be a big change under-the-covers. Things that
> scare me include:
> * do we need to re-implement for catalog-items the same
> dependency/version rules as OSGi uses for class-loading? (+)
Some of this is already done: we call to OSGi class loaders to find
resources, and we track search paths. If we're resolving types we don't
currently use the search path; so we need to write something to prefer
catalog items whose containingBundle matches something in the search
path. We probably also need to augment the search path with
dependencies specified through "require-bundle"; I think we currently
don't, and we'd need to grab the version OSGi uses. However I don't
think we need to implement much more than that.
> * how much will backwards compatibility with historic persisted state
> make things hard for us.
For now we're talking about _preferring_ things in search path. We're
not restricting what can be loaded. So persisted state should work the
same; you might wind up with a different version of an item in ambiguous
cases (but probably you'll get the _right_ version in the new world
whereas you'd be getting the _wrong_ version in the old world).
We will be supporting "legacy" catalog items that don't come from a
bundle until either those are upgraded or we transparently wrap them in
a bundle on upgrade.
>
> Are there sensible ways to scope this and de-risk it? Any incremental
> milestones, which give us incremental value or de-risk it?
>
> (+) We could potentially hack something where we synthesise a
> package/resource per catalog item in our auto-generated bundles, and
> then ask OSGi where it would load that from! (But that feels like a
> bad idea, because power-users might add their own bundles directly to
> karaf.)
>
It feels incremental already. Do OSGi versions first, then require
everything comes from bundles, then we prefer bundle search paths.
> _*Business Case (aka value to end-users)*_
> It would be good to be clearer on the user-value we deliver from this
> (but please shout if others think that is already clear enough).
>
> We're making "upgrades" easier - focusing specifically on deploying a
> second version of a catalog item, which can then be used (ignoring
> upgrade of existing app instances for now).
>
> We're solving something like the DLL-hell problem (currently, if you
> don't declare explicit versions within your blueprint, then your
> blueprint might break when someone else adds a newer version of your
> dependent blueprint for their own usage). Our current workaround for
> DLL hell is impractical: hardcode explicit version numbers of your
> dependencies in the `type: name:version` (no support for version
> ranges etc, and it would be burried deep within the yaml files).
>
> We're making it easier to deploy different versions of a "group of
> catalog items" (e.g. a big app that is split into several catalog
> items). Currently (if you didn't hardcode versions in every `type:
> ...` reference), deploying a 2.0.0 version would likely break a
> subsequent deployment of the 1.0.0 version, because it would default
> to the latest for each catalog reference. That is, you'd only get the
> 1.0.0 version of the main item, and then the 2.0.0 version of each
> item it referenced.
>
> We'd move away from `brooklyn.libraries` hard-coding URLs to other
> bundles, which most likely goes against the security policies of many
> enterprises!
>
> Is that a fair summary? Anything to add/change?
>
Great list. I'd say we're also making type management easier because I
can view related types within a bundle.
> Aled
Best
Alex
>
>
> On 21/06/2017 09:41, Geoff Macartney wrote:
>> hi Alex,
>>
>> that sounds fair enough. We'll want to add good documentation around
>> what
>> it means if *everything* is now a bundle (and how we wrap plain catalogs
>> into bundles on users' behalf). We can use the text of your email as a
>> starting point for the docs; it would be good to explain some of the
>> _motivation_ around the choice to make everything a bundle, rather than
>> just say what we do.
>>
>> I do like the idea of version ranges in brooklyn.libraries, but agree
>> it's
>> best to proceed in small steps so that would be a later development.
>>
>> Some more questions on implementation details
>>
>> - is it part of this plan to move bundle storage into the persistence
>> location so that HA Brooklyn nodes can find all bundles?
>> - if a catalog.bom (symbolic name xyz) includes brooklyn.libraries and
>> specifies a remote URL (which gets downloaded), what happens if we
>> delete
>> the bundle xyz - do the downloaded bundles get "cascade deleted"?
>>
>> Geoff
>>
>>
>> On Tue, 20 Jun 2017 at 17:22 Alex Heneveld
>> <al...@cloudsoftcorp.com>
>> wrote:
>>
>>> Hi Geoff
>>>
>>> Thanks. Replies to specific points:
>>>
>>> > whether we are "leaking" an implementation detail (the fact that we
>>> use an OSGI runtime)
>>>
>>> My motivation was to try to minimize this leak. The fact that we use
>>> OSGi internally for
>>> dependency isolation forces us to use OSGi versions with bundles. But
>>> most people
>>> most places use the "-qualifier" suffix not ".qualifier", so to
>>> accommodate that the
>>> proposal suggests a recommended syntax in line with what most people do
>>> ("-qualifer")
>>> which we can internally safely convert to an OSGi version when needed.
>>>
>>> There is no need for authors ever to see OSGi and the docs PR doesn't
>>> mention OSGi
>>> until an "advanced" section.
>>>
>>> > proposal would rely on Brooklyn moving exclusively to Karaf
>>>
>>> No. The bundle version format is already a problem with the use of
>>> embedded Felix
>>> (it solves the problem noted in #672 about needing to use
>>> ".qualifer" in
>>> your catalog.bom)
>>>
>>> > libraries / required-bundle / version ranges
>>>
>>> Where users want to indicate dependent bundles they currently have the
>>> choice of
>>> "brooklyn.libraries" (non-osgi) or "require-bundle" (osgi). Both work.
>>> We don't need to disable the
>>> former. I'm in two minds about whether we'd want to, at some point,
>>> but
>>> not right now of course.
>>>
>>> An argument for disabling is that it sets up a single canonical way to
>>> do things. Yes it
>>> means people have to learn OSGi but this is a fairly advanced use case
>>> and OSGi is the right tool for the job.
>>>
>>> If we didn't, it almost certainly means adding version range support in
>>> `brooklyn.libraries` which
>>> is yet more duplication of what OSGi lets us do. However it would be
>>> easier for non-OSGi users
>>> (and a lot of OSGi users I bet!) if we did invest in a nice range
>>> syntax
>>> in `brooklyn.libraries`.
>>>
>>> Note `brooklyn.libraries` is the only place where you can specify a URL
>>> for a dependent bundle;
>>> the OSGi manifest doesn't let you do this (though if using Karaf you
>>> can
>>> configure it to look in a set of repos,
>>> which is possibly).
>>>
>>> It boils down to how we expect people to manage the installation of
>>> multiple catalog bundles
>>> (URLs v configured repos v something else ... maybe KAR archives which
>>> could easily be supported
>>> but they're wasteful if the same dependency is included in many KARs).
>>>
>>> Not sure where we'll want to go with this but it's not something we
>>> need
>>> to solve for this proposal.
>>>
>>>
>>> HTH
>>>
>>> Best
>>> Alex
>>>
>>>
>>> On 20/06/2017 17:00, Geoff Macartney wrote:
>>>> hi Alex,
>>>>
>>>> In general I think this is a good direction to go in, some
>>>> questions and
>>>> thoughts below.
>>>>
>>>> One concern is whether we are "leaking" an implementation detail (the
>>> fact
>>>> that we
>>>> use an OSGI runtime) into the specification of what Brooklyn is. I
>>>> think
>>>> it is good to
>>>> aim for the improved modularity of having a bundle of catalog
>>>> definitions
>>>> and related
>>>> resources, all at a given version and added or removed as a unit. I'm
>>> less
>>>> sure we
>>>> want to advertise that all Brooklyn definitions must be contained
>>>> in OSGI
>>>> bundles,
>>>> even if we will wrap them for you. This seems to impose an additional
>>>> requirement
>>>> on blueprint authors (learn OSGI) that we don't really want to impose?
>>>>
>>>> As an aside, I'm assuming that your proposal would rely on Brooklyn
>>> moving
>>>> exclusively to the Karaf launcher, is that the case?
>>>>
>>>> Specific notes inline below.
>>>>
>>>> Geoff
>>>>
>>>> On Mon, 19 Jun 2017 at 17:39 Alex Heneveld <
>>> alex.heneveld@cloudsoftcorp.com>
>>>> wrote:
>>>>
>>>>> Hi All-
>>>>>
>>>>> With bundles now being neatly supported for add and remove, I
>>>>> think it
>>>>> makes sense to use make the "bundle" a first-class concept in
>>>>> managing a
>>>>> brooklyn catalog (type registry).
>>>>>
>>>>> The basic idea is that users will work with ZIPs (or directories if
>>>>> using the br CLI), where a "catalog.bom" is in the root; this is a
>>>>> bundle, and operations like add/remove/update apply to the bundle.
>>>>> Increasingly people are doing this for additions, so it seems pretty
>>>>> obvious that delete should follow suit, and something like a
>>>>> "/bundles"
>>>>> REST endpoint is appropriate.
>>>>>
>>>> I do like this; as part of this we should re-instate the automatic
>>> scanning
>>>> of bundles for catalog.bom (feature FEATURE_LOAD_BUNDLE_CATALOG_BOM).
>>>> Now that catalog items can tell you their containing bundle
>>>> (getContainingBundle())
>>>> it should be possible to make this work with rebind.
>>>>
>>>>
>>>>
>>>>> You can still upload just a catalog.bom file; in this proposal it
>>>>> will
>>>>> be wrapped in a bundle (internally, hidden from user). A bundle name
>>>>> and version will have to be inferred/guessed for backwards
>>>>> compatibility, and probably we should deprecate supplying a BOM which
>>>>> doesn't define a bundle name and version.
>>>>>
>>>> +1
>>>>
>>>>
>>>>> If a ZIP/folder is supplied, it doesn't have to contain any OSGi
>>>>> bundle
>>>>> definition; that call all be inferred. If a MANIFEST.MF is supplied,
>>>>> any bundle name/version given there must of course match that in the
>>>>> catalog.bom file.
>>>>
>>>>
>>>>> If dependent bundles are declared, they are obeyed at
>>>>> runtime and can be used to find things (pretty much identically to
>>>>> how
>>>>> we treat "libraries" in the catalog.bom.)
>>>>>
>>>> Not so sure about that last paragraph - again this is related to
>>>> leaking
>>>> the use
>>>> of OSGI. This would allow someone to write a catalog.bom that
>>>> would ONLY
>>>> work when supplied along with a related manifest (to specify certain
>>>> dependencies
>>>> and so set up the search paths). Do we want to allow that, or
>>>> should all
>>>> definitions
>>>> required for entities in a catalog be supplied in the catalog.bom
>>>> (or its
>>>> included
>>>> catalog definitions).
>>>>
>>>>
>>>>
>>>>> * Search Paths - Bundles fix an issue where types have implicit
>>>>> version
>>>>> dependencies. Consider an "acme-cluster" of "acme-node" entities,
>>>>> defined in a bundle. If I release a 1.0 version of this, and then
>>>>> a 2.0
>>>>> version, I'd probably expect the 1.0 acme-cluster to load 1.0
>>>>> acme-node
>>>>> entities, and same for 2.0. But currently any reference to
>>>>> "acme-node"
>>>>> takes the latest version, even if there is a different version in the
>>>>> same bundle. Shifting to bundles, I think we should change this, so
>>>>> that versions of items in the same bundle (or on the OSGi or library
>>>>> search path for that bundle) are preferred over more recent
>>>>> versions in
>>>>> the repository.
>>>> +1 this sounds like a better match to what blueprint authors intend
>>>>
>>>>
>>>>> (And since we will follow OSGi loading, we can also, if
>>>>> we need to, use OSGi config to say that "acme" bundle requires a
>>>>> given
>>>>> version _range_ for some other bundle, to have more control over
>>>>> dependency versions when required.)
>>>>>
>>>> again, how "use OSGI config" - there may be value in specifying
>>>> ranges
>>> but
>>>> we wouldn't want users to have to use OSGI declarations to achieve it.
>>>>
>>>>
>>>>
>>>>> For most users and blueprints this will cause no changes. However
>>>>> there
>>>>> are some changes to note:
>>>>>
>>>>> * If people have been sloppy with versions or metadata, in most cases
>>>>> they will get a warning
>>>>> * In some cases if uploading a ZIP, the catalog.bom may need to be
>>>>> edited (the upload will give a clear error, and it should not expect
>>>>> rebind)
>>>>> * When a type is referenced with no version, the loading semantics
>>>>> may
>>>>> change if that type is in the same bundle (this is probably what the
>>>>> user expects)
>>>>>
>>>>> Does anyone oppose any of the above, or have further questions?
>>>>>
>>>>> Best
>>>>> Alex
>>>>>
>>>>>
>>>>> [1] https://github.com/apache/brooklyn-server/pull/672
>>>>>
>>>>>
>>>>>
>>>
>
>
Re: Bundles in Brooklyn
Posted by Richard Downer <ri...@apache.org>.
Hi Aled,
On 22 June 2017 at 18:51, Aled Sage <al...@gmail.com> wrote:
> _*Business Case (aka value to end-users)*_
> It would be good to be clearer on the user-value we deliver from this (but
> please shout if others think that is already clear enough).
>
I have a fairly simple use case. At the moment (and I may be using Brooklyn
wrongly) when I'm incrementally developing blueprints and their related
resources in my IDE, I tell Brooklyn to install a BOM file which looks a
bit like this:
brooklyn.catalog:
brooklyn.libraries:
- name: foo.bar.MyApp
version: 0.1.0.SNAPSHOT
url: mvn:foo.bar/myapp/0.1.0-SNAPSHOT
items:
- "classpath://x.bom"
- "classpath://y.bom"
- "classpath://z.bom"
This installs the bundle and adds multiple BOMs, which in turn install
multiple catalog items.
But if I want to clean up my development work, I have to uninstall all the
catalog items manually, as well as my bundle.
Now the ZIP support is making this easier, but I would like it to be easier
still - I want to install the ZIP, and have all the catalog items magically
appear. I also want to be able to uninstall the ZIP, and have all the
catalog items disappear, everything cleaned up! And I also want this to be
true for Java entities inside OSGI bundles, not just ZIPs.
It's about have a single artifact which is treated as a unit, and
JustWorks(tm) - install it and everything is ready in one go, uninstall it
and everything is cleaned up in one go.
Richard.
Re: Bundles in Brooklyn
Posted by Aled Sage <al...@gmail.com>.
+1 to this approach, but I suspect there's a lot more to discuss/plan.
_*Leaking OSGi*_
I agree with Geoff's worry of us "leaking OSGi". However, we should just
"leak" the semantics of bundle dependencies and versioning, which need
to be explained to the user. I think we can avoid exposing the OSGi
syntax anywhere (except for power-users who want it).
_*`brooklyn.libraries` versus `package-imports analogy*_
OSGi recommends the use of package-imports, rather than explicitly
depending on specific bundles.
The analogy for us would be to depend on other catalog item
name:versionRange (e.g. in a `brooklyn.imports` section, or whatever).
We'd then figure out which bundle to get those from using the same rules
as OSGi package-imports.
Is it worth exploring further that option, rather than going down the
`brooklyn.libraries` route?
_*References Between Bundles*_
I don't think long-term the `brooklyn.libraries` should have URLs of
other bundles. It could give the name:versionRange of other bundles
(which assumes they have somehow already been added to Brooklyn).
(Reasoning is that other smart folk don't embed the URLs, presumably
because they want people to chose which repo things will come from -
e.g. enterprise private repo, versus maven central, etc).
_*Deploying Multiple Bundles*_
I lean towards something analogous to the kar file, but maybe lighter
weight rather than introducing anything like "features".
Really we just need a list of bundle URLs. If someone then deletes one
of those bundles, which breaks the other bundles deployed along with it,
then tough!
_*Blueprint Validation*_
Longer term, it would be nice if we further improved our blueprint
validation to ensure all references in a catalog item being added are
actually resolvable.
Adding to Brooklyn the ability to add a list-of-bundles will no doubt
make that harder!
_*Incremental Steps?*_
This might well prove to be a big change under-the-covers. Things that
scare me include:
* do we need to re-implement for catalog-items the same
dependency/version rules as OSGi uses for class-loading? (+)
* how much will backwards compatibility with historic persisted state
make things hard for us.
Are there sensible ways to scope this and de-risk it? Any incremental
milestones, which give us incremental value or de-risk it?
(+) We could potentially hack something where we synthesise a
package/resource per catalog item in our auto-generated bundles, and
then ask OSGi where it would load that from! (But that feels like a bad
idea, because power-users might add their own bundles directly to karaf.)
_*Business Case (aka value to end-users)*_
It would be good to be clearer on the user-value we deliver from this
(but please shout if others think that is already clear enough).
We're making "upgrades" easier - focusing specifically on deploying a
second version of a catalog item, which can then be used (ignoring
upgrade of existing app instances for now).
We're solving something like the DLL-hell problem (currently, if you
don't declare explicit versions within your blueprint, then your
blueprint might break when someone else adds a newer version of your
dependent blueprint for their own usage). Our current workaround for DLL
hell is impractical: hardcode explicit version numbers of your
dependencies in the `type: name:version` (no support for version ranges
etc, and it would be burried deep within the yaml files).
We're making it easier to deploy different versions of a "group of
catalog items" (e.g. a big app that is split into several catalog
items). Currently (if you didn't hardcode versions in every `type: ...`
reference), deploying a 2.0.0 version would likely break a subsequent
deployment of the 1.0.0 version, because it would default to the latest
for each catalog reference. That is, you'd only get the 1.0.0 version of
the main item, and then the 2.0.0 version of each item it referenced.
We'd move away from `brooklyn.libraries` hard-coding URLs to other
bundles, which most likely goes against the security policies of many
enterprises!
Is that a fair summary? Anything to add/change?
Aled
On 21/06/2017 09:41, Geoff Macartney wrote:
> hi Alex,
>
> that sounds fair enough. We'll want to add good documentation around what
> it means if *everything* is now a bundle (and how we wrap plain catalogs
> into bundles on users' behalf). We can use the text of your email as a
> starting point for the docs; it would be good to explain some of the
> _motivation_ around the choice to make everything a bundle, rather than
> just say what we do.
>
> I do like the idea of version ranges in brooklyn.libraries, but agree it's
> best to proceed in small steps so that would be a later development.
>
> Some more questions on implementation details
>
> - is it part of this plan to move bundle storage into the persistence
> location so that HA Brooklyn nodes can find all bundles?
> - if a catalog.bom (symbolic name xyz) includes brooklyn.libraries and
> specifies a remote URL (which gets downloaded), what happens if we delete
> the bundle xyz - do the downloaded bundles get "cascade deleted"?
>
> Geoff
>
>
> On Tue, 20 Jun 2017 at 17:22 Alex Heneveld <al...@cloudsoftcorp.com>
> wrote:
>
>> Hi Geoff
>>
>> Thanks. Replies to specific points:
>>
>> > whether we are "leaking" an implementation detail (the fact that we
>> use an OSGI runtime)
>>
>> My motivation was to try to minimize this leak. The fact that we use
>> OSGi internally for
>> dependency isolation forces us to use OSGi versions with bundles. But
>> most people
>> most places use the "-qualifier" suffix not ".qualifier", so to
>> accommodate that the
>> proposal suggests a recommended syntax in line with what most people do
>> ("-qualifer")
>> which we can internally safely convert to an OSGi version when needed.
>>
>> There is no need for authors ever to see OSGi and the docs PR doesn't
>> mention OSGi
>> until an "advanced" section.
>>
>> > proposal would rely on Brooklyn moving exclusively to Karaf
>>
>> No. The bundle version format is already a problem with the use of
>> embedded Felix
>> (it solves the problem noted in #672 about needing to use ".qualifer" in
>> your catalog.bom)
>>
>> > libraries / required-bundle / version ranges
>>
>> Where users want to indicate dependent bundles they currently have the
>> choice of
>> "brooklyn.libraries" (non-osgi) or "require-bundle" (osgi). Both work.
>> We don't need to disable the
>> former. I'm in two minds about whether we'd want to, at some point, but
>> not right now of course.
>>
>> An argument for disabling is that it sets up a single canonical way to
>> do things. Yes it
>> means people have to learn OSGi but this is a fairly advanced use case
>> and OSGi is the right tool for the job.
>>
>> If we didn't, it almost certainly means adding version range support in
>> `brooklyn.libraries` which
>> is yet more duplication of what OSGi lets us do. However it would be
>> easier for non-OSGi users
>> (and a lot of OSGi users I bet!) if we did invest in a nice range syntax
>> in `brooklyn.libraries`.
>>
>> Note `brooklyn.libraries` is the only place where you can specify a URL
>> for a dependent bundle;
>> the OSGi manifest doesn't let you do this (though if using Karaf you can
>> configure it to look in a set of repos,
>> which is possibly).
>>
>> It boils down to how we expect people to manage the installation of
>> multiple catalog bundles
>> (URLs v configured repos v something else ... maybe KAR archives which
>> could easily be supported
>> but they're wasteful if the same dependency is included in many KARs).
>>
>> Not sure where we'll want to go with this but it's not something we need
>> to solve for this proposal.
>>
>>
>> HTH
>>
>> Best
>> Alex
>>
>>
>> On 20/06/2017 17:00, Geoff Macartney wrote:
>>> hi Alex,
>>>
>>> In general I think this is a good direction to go in, some questions and
>>> thoughts below.
>>>
>>> One concern is whether we are "leaking" an implementation detail (the
>> fact
>>> that we
>>> use an OSGI runtime) into the specification of what Brooklyn is. I think
>>> it is good to
>>> aim for the improved modularity of having a bundle of catalog definitions
>>> and related
>>> resources, all at a given version and added or removed as a unit. I'm
>> less
>>> sure we
>>> want to advertise that all Brooklyn definitions must be contained in OSGI
>>> bundles,
>>> even if we will wrap them for you. This seems to impose an additional
>>> requirement
>>> on blueprint authors (learn OSGI) that we don't really want to impose?
>>>
>>> As an aside, I'm assuming that your proposal would rely on Brooklyn
>> moving
>>> exclusively to the Karaf launcher, is that the case?
>>>
>>> Specific notes inline below.
>>>
>>> Geoff
>>>
>>> On Mon, 19 Jun 2017 at 17:39 Alex Heneveld <
>> alex.heneveld@cloudsoftcorp.com>
>>> wrote:
>>>
>>>> Hi All-
>>>>
>>>> With bundles now being neatly supported for add and remove, I think it
>>>> makes sense to use make the "bundle" a first-class concept in managing a
>>>> brooklyn catalog (type registry).
>>>>
>>>> The basic idea is that users will work with ZIPs (or directories if
>>>> using the br CLI), where a "catalog.bom" is in the root; this is a
>>>> bundle, and operations like add/remove/update apply to the bundle.
>>>> Increasingly people are doing this for additions, so it seems pretty
>>>> obvious that delete should follow suit, and something like a "/bundles"
>>>> REST endpoint is appropriate.
>>>>
>>> I do like this; as part of this we should re-instate the automatic
>> scanning
>>> of bundles for catalog.bom (feature FEATURE_LOAD_BUNDLE_CATALOG_BOM).
>>> Now that catalog items can tell you their containing bundle
>>> (getContainingBundle())
>>> it should be possible to make this work with rebind.
>>>
>>>
>>>
>>>> You can still upload just a catalog.bom file; in this proposal it will
>>>> be wrapped in a bundle (internally, hidden from user). A bundle name
>>>> and version will have to be inferred/guessed for backwards
>>>> compatibility, and probably we should deprecate supplying a BOM which
>>>> doesn't define a bundle name and version.
>>>>
>>> +1
>>>
>>>
>>>> If a ZIP/folder is supplied, it doesn't have to contain any OSGi bundle
>>>> definition; that call all be inferred. If a MANIFEST.MF is supplied,
>>>> any bundle name/version given there must of course match that in the
>>>> catalog.bom file.
>>>
>>>
>>>> If dependent bundles are declared, they are obeyed at
>>>> runtime and can be used to find things (pretty much identically to how
>>>> we treat "libraries" in the catalog.bom.)
>>>>
>>> Not so sure about that last paragraph - again this is related to leaking
>>> the use
>>> of OSGI. This would allow someone to write a catalog.bom that would ONLY
>>> work when supplied along with a related manifest (to specify certain
>>> dependencies
>>> and so set up the search paths). Do we want to allow that, or should all
>>> definitions
>>> required for entities in a catalog be supplied in the catalog.bom (or its
>>> included
>>> catalog definitions).
>>>
>>>
>>>
>>>> * Search Paths - Bundles fix an issue where types have implicit version
>>>> dependencies. Consider an "acme-cluster" of "acme-node" entities,
>>>> defined in a bundle. If I release a 1.0 version of this, and then a 2.0
>>>> version, I'd probably expect the 1.0 acme-cluster to load 1.0 acme-node
>>>> entities, and same for 2.0. But currently any reference to "acme-node"
>>>> takes the latest version, even if there is a different version in the
>>>> same bundle. Shifting to bundles, I think we should change this, so
>>>> that versions of items in the same bundle (or on the OSGi or library
>>>> search path for that bundle) are preferred over more recent versions in
>>>> the repository.
>>> +1 this sounds like a better match to what blueprint authors intend
>>>
>>>
>>>> (And since we will follow OSGi loading, we can also, if
>>>> we need to, use OSGi config to say that "acme" bundle requires a given
>>>> version _range_ for some other bundle, to have more control over
>>>> dependency versions when required.)
>>>>
>>> again, how "use OSGI config" - there may be value in specifying ranges
>> but
>>> we wouldn't want users to have to use OSGI declarations to achieve it.
>>>
>>>
>>>
>>>> For most users and blueprints this will cause no changes. However there
>>>> are some changes to note:
>>>>
>>>> * If people have been sloppy with versions or metadata, in most cases
>>>> they will get a warning
>>>> * In some cases if uploading a ZIP, the catalog.bom may need to be
>>>> edited (the upload will give a clear error, and it should not expect
>>>> rebind)
>>>> * When a type is referenced with no version, the loading semantics may
>>>> change if that type is in the same bundle (this is probably what the
>>>> user expects)
>>>>
>>>> Does anyone oppose any of the above, or have further questions?
>>>>
>>>> Best
>>>> Alex
>>>>
>>>>
>>>> [1] https://github.com/apache/brooklyn-server/pull/672
>>>>
>>>>
>>>>
>>
Re: Bundles in Brooklyn
Posted by Geoff Macartney <ge...@cloudsoft.io>.
hi Alex,
that sounds fair enough. We'll want to add good documentation around what
it means if *everything* is now a bundle (and how we wrap plain catalogs
into bundles on users' behalf). We can use the text of your email as a
starting point for the docs; it would be good to explain some of the
_motivation_ around the choice to make everything a bundle, rather than
just say what we do.
I do like the idea of version ranges in brooklyn.libraries, but agree it's
best to proceed in small steps so that would be a later development.
Some more questions on implementation details
- is it part of this plan to move bundle storage into the persistence
location so that HA Brooklyn nodes can find all bundles?
- if a catalog.bom (symbolic name xyz) includes brooklyn.libraries and
specifies a remote URL (which gets downloaded), what happens if we delete
the bundle xyz - do the downloaded bundles get "cascade deleted"?
Geoff
On Tue, 20 Jun 2017 at 17:22 Alex Heneveld <al...@cloudsoftcorp.com>
wrote:
>
> Hi Geoff
>
> Thanks. Replies to specific points:
>
> > whether we are "leaking" an implementation detail (the fact that we
> use an OSGI runtime)
>
> My motivation was to try to minimize this leak. The fact that we use
> OSGi internally for
> dependency isolation forces us to use OSGi versions with bundles. But
> most people
> most places use the "-qualifier" suffix not ".qualifier", so to
> accommodate that the
> proposal suggests a recommended syntax in line with what most people do
> ("-qualifer")
> which we can internally safely convert to an OSGi version when needed.
>
> There is no need for authors ever to see OSGi and the docs PR doesn't
> mention OSGi
> until an "advanced" section.
>
> > proposal would rely on Brooklyn moving exclusively to Karaf
>
> No. The bundle version format is already a problem with the use of
> embedded Felix
> (it solves the problem noted in #672 about needing to use ".qualifer" in
> your catalog.bom)
>
> > libraries / required-bundle / version ranges
>
> Where users want to indicate dependent bundles they currently have the
> choice of
> "brooklyn.libraries" (non-osgi) or "require-bundle" (osgi). Both work.
> We don't need to disable the
> former. I'm in two minds about whether we'd want to, at some point, but
> not right now of course.
>
> An argument for disabling is that it sets up a single canonical way to
> do things. Yes it
> means people have to learn OSGi but this is a fairly advanced use case
> and OSGi is the right tool for the job.
>
> If we didn't, it almost certainly means adding version range support in
> `brooklyn.libraries` which
> is yet more duplication of what OSGi lets us do. However it would be
> easier for non-OSGi users
> (and a lot of OSGi users I bet!) if we did invest in a nice range syntax
> in `brooklyn.libraries`.
>
> Note `brooklyn.libraries` is the only place where you can specify a URL
> for a dependent bundle;
> the OSGi manifest doesn't let you do this (though if using Karaf you can
> configure it to look in a set of repos,
> which is possibly).
>
> It boils down to how we expect people to manage the installation of
> multiple catalog bundles
> (URLs v configured repos v something else ... maybe KAR archives which
> could easily be supported
> but they're wasteful if the same dependency is included in many KARs).
>
> Not sure where we'll want to go with this but it's not something we need
> to solve for this proposal.
>
>
> HTH
>
> Best
> Alex
>
>
> On 20/06/2017 17:00, Geoff Macartney wrote:
> > hi Alex,
> >
> > In general I think this is a good direction to go in, some questions and
> > thoughts below.
> >
> > One concern is whether we are "leaking" an implementation detail (the
> fact
> > that we
> > use an OSGI runtime) into the specification of what Brooklyn is. I think
> > it is good to
> > aim for the improved modularity of having a bundle of catalog definitions
> > and related
> > resources, all at a given version and added or removed as a unit. I'm
> less
> > sure we
> > want to advertise that all Brooklyn definitions must be contained in OSGI
> > bundles,
> > even if we will wrap them for you. This seems to impose an additional
> > requirement
> > on blueprint authors (learn OSGI) that we don't really want to impose?
> >
> > As an aside, I'm assuming that your proposal would rely on Brooklyn
> moving
> > exclusively to the Karaf launcher, is that the case?
> >
> > Specific notes inline below.
> >
> > Geoff
> >
> > On Mon, 19 Jun 2017 at 17:39 Alex Heneveld <
> alex.heneveld@cloudsoftcorp.com>
> > wrote:
> >
> >> Hi All-
> >>
> >> With bundles now being neatly supported for add and remove, I think it
> >> makes sense to use make the "bundle" a first-class concept in managing a
> >> brooklyn catalog (type registry).
> >>
> >> The basic idea is that users will work with ZIPs (or directories if
> >> using the br CLI), where a "catalog.bom" is in the root; this is a
> >> bundle, and operations like add/remove/update apply to the bundle.
> >> Increasingly people are doing this for additions, so it seems pretty
> >> obvious that delete should follow suit, and something like a "/bundles"
> >> REST endpoint is appropriate.
> >>
> > I do like this; as part of this we should re-instate the automatic
> scanning
> > of bundles for catalog.bom (feature FEATURE_LOAD_BUNDLE_CATALOG_BOM).
> > Now that catalog items can tell you their containing bundle
> > (getContainingBundle())
> > it should be possible to make this work with rebind.
> >
> >
> >
> >> You can still upload just a catalog.bom file; in this proposal it will
> >> be wrapped in a bundle (internally, hidden from user). A bundle name
> >> and version will have to be inferred/guessed for backwards
> >> compatibility, and probably we should deprecate supplying a BOM which
> >> doesn't define a bundle name and version.
> >>
> > +1
> >
> >
> >> If a ZIP/folder is supplied, it doesn't have to contain any OSGi bundle
> >> definition; that call all be inferred. If a MANIFEST.MF is supplied,
> >> any bundle name/version given there must of course match that in the
> >> catalog.bom file.
> >
> >
> >
> >> If dependent bundles are declared, they are obeyed at
> >> runtime and can be used to find things (pretty much identically to how
> >> we treat "libraries" in the catalog.bom.)
> >>
> > Not so sure about that last paragraph - again this is related to leaking
> > the use
> > of OSGI. This would allow someone to write a catalog.bom that would ONLY
> > work when supplied along with a related manifest (to specify certain
> > dependencies
> > and so set up the search paths). Do we want to allow that, or should all
> > definitions
> > required for entities in a catalog be supplied in the catalog.bom (or its
> > included
> > catalog definitions).
> >
> >
> >
> >> * Search Paths - Bundles fix an issue where types have implicit version
> >> dependencies. Consider an "acme-cluster" of "acme-node" entities,
> >> defined in a bundle. If I release a 1.0 version of this, and then a 2.0
> >> version, I'd probably expect the 1.0 acme-cluster to load 1.0 acme-node
> >> entities, and same for 2.0. But currently any reference to "acme-node"
> >> takes the latest version, even if there is a different version in the
> >> same bundle. Shifting to bundles, I think we should change this, so
> >> that versions of items in the same bundle (or on the OSGi or library
> >> search path for that bundle) are preferred over more recent versions in
> >> the repository.
> >
> > +1 this sounds like a better match to what blueprint authors intend
> >
> >
> >> (And since we will follow OSGi loading, we can also, if
> >> we need to, use OSGi config to say that "acme" bundle requires a given
> >> version _range_ for some other bundle, to have more control over
> >> dependency versions when required.)
> >>
> > again, how "use OSGI config" - there may be value in specifying ranges
> but
> > we wouldn't want users to have to use OSGI declarations to achieve it.
> >
> >
> >
> >> For most users and blueprints this will cause no changes. However there
> >> are some changes to note:
> >>
> >> * If people have been sloppy with versions or metadata, in most cases
> >> they will get a warning
> >> * In some cases if uploading a ZIP, the catalog.bom may need to be
> >> edited (the upload will give a clear error, and it should not expect
> >> rebind)
> >> * When a type is referenced with no version, the loading semantics may
> >> change if that type is in the same bundle (this is probably what the
> >> user expects)
> >>
> >> Does anyone oppose any of the above, or have further questions?
> >>
> >> Best
> >> Alex
> >>
> >>
> >> [1] https://github.com/apache/brooklyn-server/pull/672
> >>
> >>
> >>
>
>
Re: Bundles in Brooklyn
Posted by Alex Heneveld <al...@cloudsoftcorp.com>.
Hi Geoff
Thanks. Replies to specific points:
> whether we are "leaking" an implementation detail (the fact that we
use an OSGI runtime)
My motivation was to try to minimize this leak. The fact that we use
OSGi internally for
dependency isolation forces us to use OSGi versions with bundles. But
most people
most places use the "-qualifier" suffix not ".qualifier", so to
accommodate that the
proposal suggests a recommended syntax in line with what most people do
("-qualifer")
which we can internally safely convert to an OSGi version when needed.
There is no need for authors ever to see OSGi and the docs PR doesn't
mention OSGi
until an "advanced" section.
> proposal would rely on Brooklyn moving exclusively to Karaf
No. The bundle version format is already a problem with the use of
embedded Felix
(it solves the problem noted in #672 about needing to use ".qualifer" in
your catalog.bom)
> libraries / required-bundle / version ranges
Where users want to indicate dependent bundles they currently have the
choice of
"brooklyn.libraries" (non-osgi) or "require-bundle" (osgi). Both work.
We don't need to disable the
former. I'm in two minds about whether we'd want to, at some point, but
not right now of course.
An argument for disabling is that it sets up a single canonical way to
do things. Yes it
means people have to learn OSGi but this is a fairly advanced use case
and OSGi is the right tool for the job.
If we didn't, it almost certainly means adding version range support in
`brooklyn.libraries` which
is yet more duplication of what OSGi lets us do. However it would be
easier for non-OSGi users
(and a lot of OSGi users I bet!) if we did invest in a nice range syntax
in `brooklyn.libraries`.
Note `brooklyn.libraries` is the only place where you can specify a URL
for a dependent bundle;
the OSGi manifest doesn't let you do this (though if using Karaf you can
configure it to look in a set of repos,
which is possibly).
It boils down to how we expect people to manage the installation of
multiple catalog bundles
(URLs v configured repos v something else ... maybe KAR archives which
could easily be supported
but they're wasteful if the same dependency is included in many KARs).
Not sure where we'll want to go with this but it's not something we need
to solve for this proposal.
HTH
Best
Alex
On 20/06/2017 17:00, Geoff Macartney wrote:
> hi Alex,
>
> In general I think this is a good direction to go in, some questions and
> thoughts below.
>
> One concern is whether we are "leaking" an implementation detail (the fact
> that we
> use an OSGI runtime) into the specification of what Brooklyn is. I think
> it is good to
> aim for the improved modularity of having a bundle of catalog definitions
> and related
> resources, all at a given version and added or removed as a unit. I'm less
> sure we
> want to advertise that all Brooklyn definitions must be contained in OSGI
> bundles,
> even if we will wrap them for you. This seems to impose an additional
> requirement
> on blueprint authors (learn OSGI) that we don't really want to impose?
>
> As an aside, I'm assuming that your proposal would rely on Brooklyn moving
> exclusively to the Karaf launcher, is that the case?
>
> Specific notes inline below.
>
> Geoff
>
> On Mon, 19 Jun 2017 at 17:39 Alex Heneveld <al...@cloudsoftcorp.com>
> wrote:
>
>> Hi All-
>>
>> With bundles now being neatly supported for add and remove, I think it
>> makes sense to use make the "bundle" a first-class concept in managing a
>> brooklyn catalog (type registry).
>>
>> The basic idea is that users will work with ZIPs (or directories if
>> using the br CLI), where a "catalog.bom" is in the root; this is a
>> bundle, and operations like add/remove/update apply to the bundle.
>> Increasingly people are doing this for additions, so it seems pretty
>> obvious that delete should follow suit, and something like a "/bundles"
>> REST endpoint is appropriate.
>>
> I do like this; as part of this we should re-instate the automatic scanning
> of bundles for catalog.bom (feature FEATURE_LOAD_BUNDLE_CATALOG_BOM).
> Now that catalog items can tell you their containing bundle
> (getContainingBundle())
> it should be possible to make this work with rebind.
>
>
>
>> You can still upload just a catalog.bom file; in this proposal it will
>> be wrapped in a bundle (internally, hidden from user). A bundle name
>> and version will have to be inferred/guessed for backwards
>> compatibility, and probably we should deprecate supplying a BOM which
>> doesn't define a bundle name and version.
>>
> +1
>
>
>> If a ZIP/folder is supplied, it doesn't have to contain any OSGi bundle
>> definition; that call all be inferred. If a MANIFEST.MF is supplied,
>> any bundle name/version given there must of course match that in the
>> catalog.bom file.
>
>
>
>> If dependent bundles are declared, they are obeyed at
>> runtime and can be used to find things (pretty much identically to how
>> we treat "libraries" in the catalog.bom.)
>>
> Not so sure about that last paragraph - again this is related to leaking
> the use
> of OSGI. This would allow someone to write a catalog.bom that would ONLY
> work when supplied along with a related manifest (to specify certain
> dependencies
> and so set up the search paths). Do we want to allow that, or should all
> definitions
> required for entities in a catalog be supplied in the catalog.bom (or its
> included
> catalog definitions).
>
>
>
>> * Search Paths - Bundles fix an issue where types have implicit version
>> dependencies. Consider an "acme-cluster" of "acme-node" entities,
>> defined in a bundle. If I release a 1.0 version of this, and then a 2.0
>> version, I'd probably expect the 1.0 acme-cluster to load 1.0 acme-node
>> entities, and same for 2.0. But currently any reference to "acme-node"
>> takes the latest version, even if there is a different version in the
>> same bundle. Shifting to bundles, I think we should change this, so
>> that versions of items in the same bundle (or on the OSGi or library
>> search path for that bundle) are preferred over more recent versions in
>> the repository.
>
> +1 this sounds like a better match to what blueprint authors intend
>
>
>> (And since we will follow OSGi loading, we can also, if
>> we need to, use OSGi config to say that "acme" bundle requires a given
>> version _range_ for some other bundle, to have more control over
>> dependency versions when required.)
>>
> again, how "use OSGI config" - there may be value in specifying ranges but
> we wouldn't want users to have to use OSGI declarations to achieve it.
>
>
>
>> For most users and blueprints this will cause no changes. However there
>> are some changes to note:
>>
>> * If people have been sloppy with versions or metadata, in most cases
>> they will get a warning
>> * In some cases if uploading a ZIP, the catalog.bom may need to be
>> edited (the upload will give a clear error, and it should not expect
>> rebind)
>> * When a type is referenced with no version, the loading semantics may
>> change if that type is in the same bundle (this is probably what the
>> user expects)
>>
>> Does anyone oppose any of the above, or have further questions?
>>
>> Best
>> Alex
>>
>>
>> [1] https://github.com/apache/brooklyn-server/pull/672
>>
>>
>>
Re: Bundles in Brooklyn
Posted by Geoff Macartney <ge...@cloudsoftcorp.com>.
hi Alex,
In general I think this is a good direction to go in, some questions and
thoughts below.
One concern is whether we are "leaking" an implementation detail (the fact
that we
use an OSGI runtime) into the specification of what Brooklyn is. I think
it is good to
aim for the improved modularity of having a bundle of catalog definitions
and related
resources, all at a given version and added or removed as a unit. I'm less
sure we
want to advertise that all Brooklyn definitions must be contained in OSGI
bundles,
even if we will wrap them for you. This seems to impose an additional
requirement
on blueprint authors (learn OSGI) that we don't really want to impose?
As an aside, I'm assuming that your proposal would rely on Brooklyn moving
exclusively to the Karaf launcher, is that the case?
Specific notes inline below.
Geoff
On Mon, 19 Jun 2017 at 17:39 Alex Heneveld <al...@cloudsoftcorp.com>
wrote:
>
> Hi All-
>
> With bundles now being neatly supported for add and remove, I think it
> makes sense to use make the "bundle" a first-class concept in managing a
> brooklyn catalog (type registry).
>
> The basic idea is that users will work with ZIPs (or directories if
> using the br CLI), where a "catalog.bom" is in the root; this is a
> bundle, and operations like add/remove/update apply to the bundle.
> Increasingly people are doing this for additions, so it seems pretty
> obvious that delete should follow suit, and something like a "/bundles"
> REST endpoint is appropriate.
>
I do like this; as part of this we should re-instate the automatic scanning
of bundles for catalog.bom (feature FEATURE_LOAD_BUNDLE_CATALOG_BOM).
Now that catalog items can tell you their containing bundle
(getContainingBundle())
it should be possible to make this work with rebind.
> You can still upload just a catalog.bom file; in this proposal it will
> be wrapped in a bundle (internally, hidden from user). A bundle name
> and version will have to be inferred/guessed for backwards
> compatibility, and probably we should deprecate supplying a BOM which
> doesn't define a bundle name and version.
>
+1
>
> If a ZIP/folder is supplied, it doesn't have to contain any OSGi bundle
> definition; that call all be inferred. If a MANIFEST.MF is supplied,
> any bundle name/version given there must of course match that in the
> catalog.bom file.
> If dependent bundles are declared, they are obeyed at
> runtime and can be used to find things (pretty much identically to how
> we treat "libraries" in the catalog.bom.)
>
Not so sure about that last paragraph - again this is related to leaking
the use
of OSGI. This would allow someone to write a catalog.bom that would ONLY
work when supplied along with a related manifest (to specify certain
dependencies
and so set up the search paths). Do we want to allow that, or should all
definitions
required for entities in a catalog be supplied in the catalog.bom (or its
included
catalog definitions).
>
> * Search Paths - Bundles fix an issue where types have implicit version
> dependencies. Consider an "acme-cluster" of "acme-node" entities,
> defined in a bundle. If I release a 1.0 version of this, and then a 2.0
> version, I'd probably expect the 1.0 acme-cluster to load 1.0 acme-node
> entities, and same for 2.0. But currently any reference to "acme-node"
> takes the latest version, even if there is a different version in the
> same bundle. Shifting to bundles, I think we should change this, so
> that versions of items in the same bundle (or on the OSGi or library
> search path for that bundle) are preferred over more recent versions in
> the repository.
+1 this sounds like a better match to what blueprint authors intend
> (And since we will follow OSGi loading, we can also, if
> we need to, use OSGi config to say that "acme" bundle requires a given
> version _range_ for some other bundle, to have more control over
> dependency versions when required.)
>
again, how "use OSGI config" - there may be value in specifying ranges but
we wouldn't want users to have to use OSGI declarations to achieve it.
>
> For most users and blueprints this will cause no changes. However there
> are some changes to note:
>
> * If people have been sloppy with versions or metadata, in most cases
> they will get a warning
> * In some cases if uploading a ZIP, the catalog.bom may need to be
> edited (the upload will give a clear error, and it should not expect
> rebind)
> * When a type is referenced with no version, the loading semantics may
> change if that type is in the same bundle (this is probably what the
> user expects)
>
> Does anyone oppose any of the above, or have further questions?
>
> Best
> Alex
>
>
> [1] https://github.com/apache/brooklyn-server/pull/672
>
>
>
Re: Bundles in Brooklyn
Posted by Alex Heneveld <al...@cloudsoftcorp.com>.
Richard-
Good points. To your questions:
> Is it necessary to always wrap stuff in a bundle?
No, and currently we don't. But I think the model becomes clearer if we
do, all catalog items (registered types) are contained in a bundle.
Note that this gives us a way to preserve an uploaded `catalog.bom`
which currently we do not have.
> Can the version number be omitted from all places except once?
Yes, I think this should be the case.
> bundle [of] multiple items ... now be treated as a unit?
I think /bundles is probably a separate REST endpoint with list, list
contents, and delete supported. In the catalog UI I imagine the
containing bundle is shown when you look at a type, you can click it to
see details of that bundle including contents, and there is a way to
list bundles.
> attempt to delete one [item] ... permitted?
As you say for legacy compatibility we have to. However I'd like to
deprecate this once everything is in a bundle and we support bundle
deletion. Once item deletion is done, we can remove the persistence of
individual catalog items; persisting the bundles is sufficient to
persist the catalog (reading BOMs from bundles on rebind). This
simplifies things which is always good. It also means if we want a
"don't delete things in use" check, and when we do upgrades, it suffices
to look for references to a bundle rather than references to specific types.
Also, I think the search path should always contain bundles. That gives
us a way to to do the above checks. Currently it contains items, and
for a transitional period it may need to contain both, but we should
de-empahsise catalog items for search purposes.
We should also deprecate the use of `libraries` anywhere other than the
top-level in catalog.bom. That makes the dependency tree simpler (it's
just among bundles, delegated to OSGi), and I'm pretty sure there's no
use case for the complexity.
Best
Alex
On 20/06/2017 11:17, Richard Downer wrote:
> Alex,
>
> On 19 June 2017 at 17:39, Alex Heneveld <al...@cloudsoftcorp.com>
> wrote:
>
>> With bundles now being neatly supported for add and remove, I think it
>> makes sense to use make the "bundle" a first-class concept in managing a
>> brooklyn catalog (type registry).
>>
> A great proposal. The blueprint developer stories in Brooklyn (which is
> where I am at the moment) are coming together really well recently, but the
> relationship between bundles and catalog items is an odd one, and it'd be
> great to get a consistent story around this.
>
>
>> The basic idea is that users will work with ZIPs (or directories if using
>> the br CLI), where a "catalog.bom" is in the root; this is a bundle, and
>> operations like add/remove/update apply to the bundle. Increasingly people
>> are doing this for additions, so it seems pretty obvious that delete should
>> follow suit, and something like a "/bundles" REST endpoint is appropriate.
>>
>> You can still upload just a catalog.bom file; in this proposal it will be
>> wrapped in a bundle (internally, hidden from user). A bundle name and
>> version will have to be inferred/guessed for backwards compatibility, and
>> probably we should deprecate supplying a BOM which doesn't define a bundle
>> name and version.
>>
> Is it necessary to always wrap stuff in a bundle? Could catalog items not
> exist independently of a bundle? (I'm not expressing an opinion about if
> this is a good or bad idea, just wondering what your thinking is.)
>
>
>> * Versions - It would normally make sense for everything in a bundle to
>> have the same version number, viz. the version number of the bundle. We
>> don't need to enforce this, but I'm thinking we should deprecate and WARN
>> if this is the case because I don't think there's any compelling reason to
>> do otherwise.
>
> +1. Can the version number be omitted from all places except once in either
> catalog.bom or MANIFEST.MF?
>
>
>> * Search Paths - Bundles fix an issue where types have implicit version
>> dependencies. Consider an "acme-cluster" of "acme-node" entities, defined
>> in a bundle. If I release a 1.0 version of this, and then a 2.0 version,
>> I'd probably expect the 1.0 acme-cluster to load 1.0 acme-node entities,
>> and same for 2.0. But currently any reference to "acme-node" takes the
>> latest version, even if there is a different version in the same bundle.
>> Shifting to bundles, I think we should change this, so that versions of
>> items in the same bundle (or on the OSGi or library search path for that
>> bundle) are preferred over more recent versions in the repository.
>
> +1
>
>
>> Does anyone oppose any of the above, or have further questions?
>>
> It all sounds good, and the compromises acceptable. A further question:
>
> If a bundle contains multiple items, would these items now be treated as a
> unit in the catalog? Say I upload a bundle which contains a couple of
> items, then use the `br` tool or Brooklyn web UI to inspect the individual
> items, and attempt to delete one. Would this be permitted? Ideally, no,
> this would not be permitted, but that would be another change in behaviour
> (and a breaking API change).
>
> Cheers
> Richard.
>
Re: Bundles in Brooklyn
Posted by Richard Downer <ri...@apache.org>.
Alex,
On 19 June 2017 at 17:39, Alex Heneveld <al...@cloudsoftcorp.com>
wrote:
> With bundles now being neatly supported for add and remove, I think it
> makes sense to use make the "bundle" a first-class concept in managing a
> brooklyn catalog (type registry).
>
A great proposal. The blueprint developer stories in Brooklyn (which is
where I am at the moment) are coming together really well recently, but the
relationship between bundles and catalog items is an odd one, and it'd be
great to get a consistent story around this.
> The basic idea is that users will work with ZIPs (or directories if using
> the br CLI), where a "catalog.bom" is in the root; this is a bundle, and
> operations like add/remove/update apply to the bundle. Increasingly people
> are doing this for additions, so it seems pretty obvious that delete should
> follow suit, and something like a "/bundles" REST endpoint is appropriate.
>
> You can still upload just a catalog.bom file; in this proposal it will be
> wrapped in a bundle (internally, hidden from user). A bundle name and
> version will have to be inferred/guessed for backwards compatibility, and
> probably we should deprecate supplying a BOM which doesn't define a bundle
> name and version.
>
Is it necessary to always wrap stuff in a bundle? Could catalog items not
exist independently of a bundle? (I'm not expressing an opinion about if
this is a good or bad idea, just wondering what your thinking is.)
> * Versions - It would normally make sense for everything in a bundle to
> have the same version number, viz. the version number of the bundle. We
> don't need to enforce this, but I'm thinking we should deprecate and WARN
> if this is the case because I don't think there's any compelling reason to
> do otherwise.
+1. Can the version number be omitted from all places except once in either
catalog.bom or MANIFEST.MF?
> * Search Paths - Bundles fix an issue where types have implicit version
> dependencies. Consider an "acme-cluster" of "acme-node" entities, defined
> in a bundle. If I release a 1.0 version of this, and then a 2.0 version,
> I'd probably expect the 1.0 acme-cluster to load 1.0 acme-node entities,
> and same for 2.0. But currently any reference to "acme-node" takes the
> latest version, even if there is a different version in the same bundle.
> Shifting to bundles, I think we should change this, so that versions of
> items in the same bundle (or on the OSGi or library search path for that
> bundle) are preferred over more recent versions in the repository.
+1
> Does anyone oppose any of the above, or have further questions?
>
It all sounds good, and the compromises acceptable. A further question:
If a bundle contains multiple items, would these items now be treated as a
unit in the catalog? Say I upload a bundle which contains a couple of
items, then use the `br` tool or Brooklyn web UI to inspect the individual
items, and attempt to delete one. Would this be permitted? Ideally, no,
this would not be permitted, but that would be another change in behaviour
(and a breaking API change).
Cheers
Richard.