Re: Uploading ZIPs for a better dev workflow

[Aled Sage <al...@gmail.com>]

Hi all,

Discussion of the REST api kicked off again in 
https://github.com/apache/brooklyn-server/pull/485#issuecomment-283280366:

    Alex wrote:

    Requiring a MANIFEST.MF makes the input strongly java-centric; I'd
    like to appeal to people who write YAML blueprints with co-bundled
    scripts and images. They'd wonder why they can't simply make and
    upload a ZIP. They could probably be persuaded to supply a name and
    optional version on a CLI or UI, and understand why it is needed
    (hence supporting those args) but it would not be idiomatic to
    anyone but a java programmer to create a META-INF/ dir with a
    MANIFEST.MF and its syntax. Meanwhile it is very easy for us to
    convert the ZIP to a JAR on the server. Feels like uncontroversial
    good UX.

    OTOH for a java programmer a MANIFEST.MF is natural, and they'd want
    to drop the name/version args if they are in that file, and I see no
    reason to forbid that pattern.

I agree with Alex, that we should not require a META-INF/MANIFEST.MF. As 
for Geoff's suggestion that we could auto-generate a manifest in the 
`br` CLI, I'd prefer a more general solution that works for users of the 
REST api as well (i.e. doing it server-side).

---
Svet suggested that the catalog.bom could give the symbolic name + 
version, via additional metadata in that file.

What I really like about that is it's in version control, becuase it's 
in the file/repo. If alternatively the name/version are just passed as 
REST api parameters, then it's not in version control (and is more 
susceptible to typos etc).

I'm not sure what we'd want this to look like. As a straw man:

    brooklyn.catalog:
       manifest:
         symbolic_name: com.example.Foo
         version: 1.0.0-SNAPSHOT
       items:
       - ...

If the user built their own real OSGi bundle, then they wouldn't need to 
include this "manifest" section. If they did include that and it 
contradicted the OSGi bundle's manifest, then we'd fail-fast.

(Note this reminds me of the (unrelated) metadata described in 
https://github.com/brooklyncentral/blueprint-repository- there is a 
"publish" block that can be added to the catalog.bom.)

---
With Svet's suggestion, if there was no manifest section/file, then 
should we auto-generate something from the item(s) in the catalog.bom?

I can see how that could easily work for a .bom file that has a single 
item (e.g. we use the catalog item's id + version, possibly with a 
special prefix in the symbolic name to avoid conflicts).

However, if there are multiple items then it would get trickier.

I'm inclined to say that for a minimal viable product (MVP) we can 
insist on the "manifest" section in the catalog.bom.

Aled


On 20/12/2016 16:34, Svetoslav Neykov wrote:
>> Svet, if instead we tried to infer it from the catalog.bom, would we require some additional metadata within the .bom file? Or would we use the catalog item's id + version? I'm not convinced by the latter - it would mean some .bom files would work and others wouldn't (e.g. if the .bom had multiple items with different versions). Better to support the explicit approach IMO.
> I imagine it would be additional metadata. On the other hand I don't see a technical reason why we need an explicit symbolicName and version - they can be auto-generated.
>
> Svet.
>
>
>> On 20.12.2016 \u0433., at 17:50, Aled Sage <al...@gmail.com> wrote:
>>
>> Hi all,
>>
>> +1
>>
>> (D) sounds good. What version are you imagining the bundle would be, if one runs `br catalog add ~/my/project/ --name com.example.myproject`?
>>
>> ---
>> I like the idea of uploading a plain zip (rather than only supporting OSGi bundles) - that makes it simpler for non-java folk. The use of OSGi becomes a (hidden) implementation detail to many users.
>>
>> ---
>> If auto-generating the manifest, I think we need the user to be explicit about symbolic name and version. Having these supplied in the REST api call (as Alex suggests) would achieve that.
>>
>> Svet, if instead we tried to infer it from the catalog.bom, would we require some additional metadata within the .bom file? Or would we use the catalog item's id + version? I'm not convinced by the latter - it would mean some .bom files would work and others wouldn't (e.g. if the .bom had multiple items with different versions). Better to support the explicit approach IMO.
>>
>> ---
>> For E ("have a mechanism whereby deployed entities based on an affected blueprint are optionally migrated to the new code"), that feels like a separate discussion. It could equally apply to a pure YAML .bom file that has been added to the catalog.
>>
>> I suggest we discuss that in a separate email thread.
>>
>> ---
>> For (G), it's an interesting suggestion from Svet to make use of Karaf Cellar for HA nodes. I'm hesitant (e.g. if restarting a standalone Brooklyn node whose VM has died, then it adds big additional requirements for what constitutes the "persisted state"). On the other hand, it's good to use well-established technologies rather than re-inventing things!
>>
>> An alternative ("pure brooklyn") approach could be to write the bundle to persisted state; on rebind, we'd install + activate those bundles.
>>
>> ---
>> For "catalogGroupId", I agree with Svet that in the initial use-case this can be an implementation detail.
>>
>> It could be set as the bundle's symbolic name + version: everything from the bundle should be deleted at once, along with the bundle.
>>
>> Longer term, I can see how exposing "catalogGroupId" to the user could support more use-cases (e.g. for several catalog items from different bundles to work together). I don't think we should try to support that yet.
>>
>> Aled
>>
>>
>> On 19/12/2016 17:19, Geoff Macartney wrote:
>>> hi Alex,
>>>
>>> this looks like a good feature to have, I shall look at the PR as soon as I
>>> can.
>>>
>>> The catalog.bom scanner feature was initially enabled by default, but we
>>> had to
>>> disable it because it turned out not to work properly with rebind. I don't
>>> think
>>> it should be a lot of work to fix that but it hasn't been something we've
>>> got round
>>> to yet.  This would be a great opportunity to look back at that.
>>>
>>> Some random thoughts:
>>>
>>> re (C), if we are going to treat the zips as bundles, my gut feel is that
>>> we
>>> should insist on a manifest and get the metadata from it.  It doesn't feel
>>> to me
>>> like it makes much sense to allow a zip file without a MANIFEST.MF but
>>> convey
>>> the intended bundle metadata to Brooklyn via HTTP headers.  And rather than
>>> infer bundle metadata I think it's better to ask users to be explicit about
>>> what
>>> their intentions are.  To make users lives easier, we could
>>> add a command to br to generate the manifest (locally) with correct syntax,
>>> so that the manifest is in the right place, rather than have br add the data
>>> to the "upload" request headers.
>>>
>>> re. (D) will be glad to have a look at it
>>>
>>> re. (E) it would certainly need to be optional - maybe keep it as an
>>> explicit
>>> separate command ('upgrade'?)
>>>
>>> (F) it does seem like a lot of work but might be nice for users who are not
>>> keen on command lines.
>>>
>>> G - I:  we'll definitely need to pay close attention to persistence and
>>> rebind;
>>> I wonder also about HA operation, are there any additional implications?
>>>
>>> (J) I think it would be good to treat all the files from a jar, sorry
>>> bundle,
>>> as an atomic group - cleaner that way perhaps than allowing delete/update
>>> of
>>> individual entries from a bundle on a piecemeal basis.  Rest support on
>>> delete
>>> catalog could warn about related catalog entries being deleted and ask for
>>> a "--force" param to confirm.
>>>
>>> Geoff
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Fri, 16 Dec 2016 at 15:24 Svetoslav Neykov <
>>> svetoslav.neykov@cloudsoftcorp.com> wrote:
>>>
>>>> +1
>>>>
>>>> Some thoughts:
>>>>    * (A) add a utility class BundleMaker
>>>>      Sounds very similar to
>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles <
>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles>
>>>>      Looking at the code it's much more focused on zip files so I guess
>>>> there's no much overlap, but worth keeping in mind
>>>>    * (C) accept bundle symbolic name and version
>>>>      Why require them at all? Could infer them from the catalog.bom in some
>>>> way - maybe require those properties to be in there. If not present are
>>>> they really needed?
>>>>    * (G) Bundles installed via this mechanism are not persisted currently &
>>>> (I) We persist the individual catalog items as YAML, so we end up with two
>>>> records
>>>>      Suggest marking the catalog items coming from bundles as
>>>> non-persistable. Then try to share the bundles between HA nodes. (Karaf
>>>> Cellar?)
>>>>    * (J) Introduce a catalogGroupId field on catalog items;
>>>>      Agree this could be useful and I like the idea of deleting the bundle
>>>> altogether with the catalog items. From user's perspective I don't see the
>>>> need for an extra field (i.e. it's an implementation detail).
>>>>
>>>> Svet.
>>>>
>>>>
>>>>> On 16.12.2016 \u0433., at 12:50, Alex Heneveld <
>>>> alex.heneveld@cloudsoftcorp.com> wrote:
>>>>> Hi Brooklyners-
>>>>>
>>>>> In the code we currently have two routes for users to install new
>>>>> blueprints:
>>>>>
>>>>> (1) upload a catalog YAML file to /v1/catalog
>>>>>
>>>>> (2) install a bundle with catalog.bom in the root
>>>>>
>>>>> The feature (2) is disabled by default, but I'd like to move towards
>>>>> enabling it.  This will make it easier to create nicely structured BOM
>>>>> files because scripts etc can be taken out of the BOM, stored as files in
>>>>> the same bundle.  (Because URLs of the form
>>>>> `classpath://scripts/install.sh`  use the bundle's classpath to resolve.)
>>>>>
>>>>> As a first step in #485 [1] I do a few things:
>>>>>
>>>>> (A) add a utility class BundleMaker that lets us create and modify
>>>>> bundles/zips, to make it easier to do things we might want to with
>>>> bundles,
>>>>> especially for testing
>>>>>
>>>>> (B) add an endpoint to the REST API which allows uploading a bundle ZIP
>>>>>
>>>>> (C) accept bundle symbolic name and version in that REST API to
>>>> facilitate
>>>>> uploading non-bundle ZIPs where the OSGi MANIFEST.MF is automatically
>>>>> generated
>>>>>
>>>>> With this PR, if you have a directory on your local file system with
>>>>> scripts and config files, and a BOM which refers to them, you can just
>>>> ZIP
>>>>> that up an upload it, specifying the bundle name so that a YAML blueprint
>>>>> author never needs to touch any java-isms.
>>>>>
>>>>> Where I see this going is a development workflow where a user can edit
>>>>> files locally and upload the ZIP to have that installed, and if they make
>>>>> changes locally they can POST it again to have catalog items updated
>>>>> (because default version is a SNAPSHOT).  We could also:
>>>>>
>>>>> (D) have `br catalog add ~/my/project/ --name my.project` create a ZIP
>>>> and
>>>>> POST it, with bundle name metadata, so essentially the user's process is
>>>>> just to run that whenever they make a change
>>>>>
>>>>> (E) have a mechanism whereby deployed entities based on an affected
>>>>> blueprint are optionally migrated to the new code, so if you've changed
>>>> an
>>>>> enricher the changes are picked up, or if say a launch.sh script has
>>>>> changed, a restart will run the new code
>>>>>
>>>>> The above are fairly straightforward programmatically (although good user
>>>>> interaction with (E) needs some thought).  So I think we can pretty
>>>> quickly
>>>>> get to a much smoother dev workflow.
>>>>>
>>>>>
>>>>> That's the highlight of this message.  You can jump to the end, unless
>>>>> you're interested in some important but low level details...
>>>>>
>>>>>
>>>>> I'm also tempted by:
>>>>>
>>>>> (F) Integration with web-based IDE and/or Brooklyn reading and writing
>>>>> straight from GitHub -- but this seems like a lot of work and I'm not
>>>>> convinced it's much better than (D) workflow-wise
>>>>>
>>>>> Before we can change (2) to be the default, or start widely using the
>>>> POST
>>>>> a ZIP feature, we need to sort out some issues to do with persistence and
>>>>> reloading:
>>>>>
>>>>> (G) Bundles installed via this mechanism are not persisted currently, so
>>>> if
>>>>> you move to a different Brooklyn using the same backing store, you'll
>>>> lose
>>>>> those bundles
>>>>>
>>>>> (H) On rebind, bundles aren't always activated when needed, meaning items
>>>>> can't be loaded
>>>>>
>>>>> (I) We persist the individual catalog items as YAML, so we end up with
>>>> two
>>>>> records \u2014 the YAML from the catalog.bom in the bundle, and the YAML
>>>>> persisted for the item.  This isn't a problem per se, but something to
>>>>> think about, and some sometimes surprising behaviour.  In particular if
>>>> you
>>>>> delete the persisted YAML, the bundle is still there so the item is no
>>>>> longer deleted after a full rebind.
>>>>>
>>>>> One idea which might be useful is:
>>>>>
>>>>> (J) Introduce a catalogGroupId field on catalog items; this will do two
>>>>> things:  if you try to delete an item with such a record, you'll be
>>>>> encouraged to delete all such items (maybe disallowed to delete an
>>>>> individual one), with the effect of deleting the bundle if it comes from
>>>> a
>>>>> bundle; and when resolving types we search first for items with the same
>>>>> catalogGroupId (so that e.g. if I install MyCluster:1.0 and MyNode:1.0 in
>>>>> the same group, the former can refer simply to "MyNode" but if I install
>>>> a
>>>>> 2.0 version of that group, the 1.0 cluster still loads the 1.0 node --
>>>> this
>>>>> has bitten people i the past)
>>>>>
>>>>> There is a related Brooklyn upgrade problem worth mentioning, which the
>>>>> above might help with, where:
>>>>>
>>>>> (K) If I migrate from Brooklyn 10 to 11 when it comes out, I'll no longer
>>>>> have certain entities that were at v10, since we don't include those; an
>>>>> upgrade could include rules that certain groupIds need to be updated, or
>>>> it
>>>>> can search and attempt to automatically apply the updates
>>>>>
>>>>>
>>>>> Quite a lot here and we don't need to solve it but I wanted to:
>>>>>
>>>>> * Share the current thinking
>>>>>
>>>>> * Get opinions on the general dev workflow suggested by (D)
>>>>>
>>>>>
>>>>> Thanks for feedback -- and if we like it help with (D) would be
>>>> appreciated!
>>>>> Best
>>>>> Alex
>>>>>
>>>>>
>>>>>
>>>>> [1] . https://github.com/apache/brooklyn-server/pull/485

Re: Uploading ZIPs for a better dev workflow

[Thomas Bouron <th...@cloudsoftcorp.com>]

I just pushed a new PR[1] that is built on top of Alex's one[2]. This
should cover all the point we discussed earlier

Best.

[1] https://github.com/apache/brooklyn-server/pull/608
[2] https://github.com/apache/brooklyn-server/pull/485

On Thu, 23 Mar 2017 at 11:27 Thomas Bouron <th...@cloudsoftcorp.com>
wrote:

> Hi Aled, all.
>
> Sounds good to me. Once the PR is merged, I'll push on another one to fix
> the other issues I found in the past 2 days.
>
> I also agree with you on 2 and 3, I'll address that.
>
> Best.
>
> On Thu, 23 Mar 2017 at 11:02 Aled Sage <al...@gmail.com> wrote:
>
> Hi Thomas, all,
>
> I think "bundle: ..." is actually a much better name to use in the
> catalog.bom than "symbolicName: ...". The latter makes sense to those
> who know OSGi, but otherwise doesn't suggest anything about "bundling".
>
> ---
>
> I think we're very close to merging [1] - further improvements can
> continue incrementally in subsequent PRs, rather than all of this having
> to be done in [1].
>
> ---
>
> There's a change compared to the mailing list discussion, I think [2,3].
> Alex has implemented it so that if "symbolicName" and "id" are used at
> the top-level of the catalog.bom, then they *must match*. This seems
> wrong to me. It's conflating the name spaces of OSGi bundles and catalog
> items. I suggest we revert back to (at least how I interpretted) the
> previous mailing list discussion: if "bundle: " is present, then that is
> used; if "bundle" is absent, then we use "id"; if both are absent, then
> we fail.
>
> Aled
>
> p.s. As an aside, we should deprecate (and soon delete) support for
> "symbolicName" in a catalog.bom - one should use id/version. Nowhere in
> our docs to we say symbolicName. It's just yet another example of us
> supporting multiple ways of doing the same thing, where it's not at all
> clear they mean the same thing, and risking confusing users.
>
> [1] https://github.com/apache/brooklyn-server/pull/485
> [2]
> https://github.com/apache/brooklyn-server/pull/485#discussion_r107144368
> [3]
> https://github.com/apache/brooklyn-server/pull/485#issuecomment-288052012
>
>
> On 23/03/2017 10:23, Geoff Macartney wrote:
> > some further thoughts:
> >
> > re. 1 I agree we should discriminate between the "symbolicName" of the
> > bundle and use of the term "symbolicName" in the definition of catalog
> ids.
> > (I'm not sure if just putting it into a "manifest" subsection will keep
> it
> > separate, because of the use of "getFirstAs" at
> >
> https://github.com/apache/brooklyn-server/blob/master/core/src/main/java/org/apache/brooklyn/core/catalog/internal/BasicBrooklynCatalog.java#L542
> > ).
> >
> > re. use of "CatalogPopulator" I think we shall want to factor apart
> > CatalogBomScanner/CatalogPopulator into separate classes, -
> > - CatalogBomScanner to be the management context service listener
> > - CatalogPopulator becomes:
> > -- CatalogBundleTracker to do the BundleTracker stuff
> > -- CatalogBundleLoader - separate out the handling of the loading of the
> > "catalog.bom" from a given bundle
> >
> > and then 485 can use CatalogBundleLoader.  We'll still need to think
> about
> > the delete side of things, and handling errors, as Thomas says.
> >
> > re. the Swagger issue, an alternative would be that we could just drop
> > "createFromYaml" and "createFromArchive" and let "createPoly" continue to
> > do as it does, handling either case; maybe best renamed
> "createFromUpload".
> >
> > Geoff
> >
> > On Thu, 23 Mar 2017 at 09:40 Thomas Bouron <
> thomas.bouron@cloudsoftcorp.com>
> > wrote:
> >
> >> Hi all.
> >>
> >> So I spent time to review and test [1] and here are the few issues I
> bumped
> >> into:
> >>
> >>     1. the use of `symbolicName` for the bundle clashes Brooklyn: the
> >>     `BasicBrooklynCatalog` watches this key to define catalog items IDs
> [2]
> >>     2. because of 1. the bundle that I'm trying to install fails. But
> upon
> >>     failure, the bundle is not uninstalled so any subsequent bundle
> addition
> >>     fail with the first error
> >>     3. we now have multiple API definitions for the same endpoint (i.e.
> >>     /v1/catalog for yaml and ZIP/JAR) While this is perfectly valid,
> Swagger
> >>     does not support it [3]. It means that the Swagger UI displays only
> the
> >>     latest API definition registered, in this case, that's the one to
> upload
> >>     ZIP/JAR.
> >>
> >> For 1. I can see 2 solutions: using a special section `manifest` where
> we
> >> put the `symbolicName` and `version`, or changing the field name to
> >> `bundle` (I prefer option 2)
> >> For 2. we need to add some logic to uninstall the bundle on failure into
> >> the `CatalogPopulator`, as per as Geoff comment[4]
> >> For 3. even the new Swagger spec (called now OpenAPI) does not support
> >> multiple definitions on the same endpoint [5]. I'm not sure if we can
> leave
> >> with that. One solution would be to split the endpoint to create
> specific
> >> one, based on the type of upload.
> >>
> >> [1] https://github.com/apache/brooklyn-server/pull/485
> >> [2]
> >>
> >>
> https://github.com/apache/brooklyn-server/blob/master/core/src/main/java/org/apache/brooklyn/core/catalog/internal/BasicBrooklynCatalog.java#L542
> >> [3] https://github.com/swagger-api/swagger-core/issues/935
> >> [4]
> >>
> https://github.com/apache/brooklyn-server/pull/485#discussion_r107482279
> >> [5] https://github.com/OAI/OpenAPI-Specification/issues/182
> >>
> >> On Mon, 20 Mar 2017 at 15:16 Thomas Bouron <
> >> thomas.bouron@cloudsoftcorp.com>
> >> wrote:
> >>
> >> Hi Brooklyners.
> >>
> >> Uploading ZIPs is really something I'm excited to see coming to life so
> I
> >> would like to help to push this forward.
> >>
> >> I've re-read this email thread and it sounds like we have a consensus on
> >> the "one true way" approach, i.e. always requiring the `symbolicName`
> and
> >> `version` when uploading ZIP/JAR (that includes Svet in his previous
> >> email).
> >>
> >> One thing not explicitly settled is the format of bom for those 2
> fields:
> >> Alex prefers to put these at the root, Aled in a `manifest` section. I
> >> don't have any strong opinion on this: I was leaning toward Aled's
> >> suggestion but the argument that says this applies to the whole bundle
> make
> >> sense so I now think putting this at the root is probably better. I can
> see
> >> that Alex already updated his PR[1] to this structure. What the people
> >> think about this?
> >>
> >> I'm going to review [1] but are we happy with this consensus? One thing
> >> that will need to be addressed is the persistence/rebind issues but this
> >> can be tackled in following PRs.
> >>
> >> Best.
> >>
> >> [1] https://github.com/apache/brooklyn-server/pull/485
> >>
> >> On Tue, 7 Mar 2017 at 16:18 Svetoslav Neykov <
> >> svetoslav.neykov@cloudsoftcorp.com> wrote:
> >>
> >> Having quickly scanned the thread I still don't see a reason why the
> >> requirement to have a consistent naming for the bundle itself which is
> not
> >> user visible? Could instead have behaviour similar to the machinNamer
> here.
> >>
> >> Slight preference for optional symbolicName when it's a jar file.
> Though I
> >> really like Geoff's idea of "one true way". The workflow to create a jar
> >> file is very different from creating a zip file (either from a folder or
> >> user supplied) so don't think they conflict.
> >>
> >> Svet.
> >>
> >>
> >>> On 7.03.2017 г., at 17:47, Alex Heneveld <
> >> alex.heneveld@cloudsoftcorp.com>
> >> wrote:
> >>>
> >>> Considering the points made I'm pretty sure it's the case that
> requiring
> >> a symbolic name and version in the BOM presents little if any
> unnecessary
> >> burden, in which case I'm persuaded of the "one true way" philosophy.  I
> >> really like it.  :)
> >>> Can anyone think of a case where it would be a burden?  My thinking is
> >> that:
> >>> * If you're uploading a ZIP of YAML and scripts and icons, you have a
> >> catalog.bom and that's a useful place to require the symname and version
> >>> * If your ZIP/JAR also has a MANIFEST.MF the symname and version will
> be
> >> present in it also; the user is duplicating that info (this is the
> "little"
> >> unnecssary burden) but it's tiny and the benefit of one true way
> dominates;
> >> we do require that they match
> >>> * If your ZIP/JAR doesn't have a BOM there's no point in uploading it
> to
> >> the catalog endpoint; either it's a JAR but this isn't meant as an
> "install
> >> arbitrary bundles" endpoint, or a non-JAR in which case what are we
> >> supposed to do with it once you upload it?
> >>> It keeps the contract simple, and the CLI very simple.   `br push
> >> /path/to/blueprint/x/`
> >>> (But Geoff as a side point I think the CLI should _not_ attempt
> >> validation of the input/version -- the server will do that anyway and no
> >> benefit in doing it twice, but more code to maintain.)
> >>> I'm further persuaded given it is in a file, we should require a
> >> version.  I was concerned about needing to supply the version every
> time on
> >> the CLI but this has gone away.
> >>> I see no reason for a separate "manifest" section.  Removing this
> >> eliminates the prospect of Aled's pathological/"perverted" example.  I
> >> favour:
> >>>     brooklyn.catalog:
> >>>        symbolicName: com.example.Foo
> >>>        version: 1.0
> >>>        items:
> >>>        - id: foo
> >>>          version: 3.0   # optional version override
> >>>          itemType: entity
> >>>          item:
> >>>            type: blah
> >>>            ...
> >>>
> >>> That is, the fields symbolic-name and version are _required_ under the
> >> `brooklyn.catalog` when POSTing a ZIP to the catalog endpoint, and are
> used
> >> to define the bundle.  The version (and icon, etc) are then inherited by
> >> default by items defined therein.  If the ZIP is a JAR which contains a
> >> MANIFEST.MF, the bundle information (symbolic name and version) must
> match
> >> exactly.
> >>> Sound right?
> >>>
> >>> Best
> >>> Alex
> >>>
> >>>
> >>> On 03/03/2017 09:59, Geoff Macartney wrote:
> >>>> It also means clients ('br' at least) can easily validate the upload
> >> even
> >>>> before execution by checking that all the manifest details are
> present.
> >>>>
> >>>> On Fri, 3 Mar 2017 at 09:57 Geoff Macartney <
> >>>> geoff.macartney@cloudsoftcorp.com> wrote:
> >>>>
> >>>>> Just on the last point of "We could also support the example below",
> >> I'd
> >>>>> say let's not even be that flexible - my feeling is that the more
> >> flexible
> >>>>> we are, the more confusing it is, and the harder to get right.  If
> >> we're
> >>>>> going to add this capability let's just have one way to do it; if you
> >> are
> >>>>> going to do a ZIP upload, you MUST have a catalog.bom, which MUST
> >> contain a
> >>>>> manifest section, which MUST have both a symbolic name and a version.
> >> That
> >>>>> way everything's explicit and very clear, every time.
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Thu, 2 Mar 2017 at 11:45 Aled Sage <al...@gmail.com> wrote:
> >>>>>
> >>>>> Geoff, all,
> >>>>>
> >>>>> I was imagining the manifest version (in the catalog.bom) to be
> >> separate
> >>>>> from the item version. The reason is that we support multiple items
> in
> >>>>> the bom that can be independently versioned.
> >>>>>
> >>>>> Somone perverted could write:
> >>>>>
> >>>>>      brooklyn.catalog:
> >>>>>         version: 1.0
> >>>>>         manifest:
> >>>>>           symbolic_name: com.example.Foo
> >>>>>           version: 2.0
> >>>>>         items:
> >>>>>         - id: foo
> >>>>>           version: 3.0
> >>>>>           itemType: entity
> >>>>>           item:
> >>>>>             type: blah
> >>>>>         - id: bar
> >>>>>           version: 4.0
> >>>>>           itemType: entity
> >>>>>           item:
> >>>>>          type: blah.blah
> >>>>>
> >>>>> Here, the "1.0" version is not used by anything; the auto-generated
> >> OSGi
> >>>>> bundle would be version 2.0; catalog item foo would be 3.0; and
> catalog
> >>>>> item bar would be 4.0.
> >>>>>
> >>>>> If we were starting from scratch, I'd be tempted to lock things down
> a
> >>>>> lot more for what we actually support. For now, it feels like asking
> >> for
> >>>>> an explicit version is a reasonable thing to do.
> >>>>>
> >>>>> ---
> >>>>> We could also support the example below, where "1.0" is used by both
> >> the
> >>>>> manifest and the item:
> >>>>>
> >>>>>      brooklyn.catalog:
> >>>>>         version: 1.0
> >>>>>         manifest:
> >>>>>           symbolic_name: com.example.Foo
> >>>>>         items:
> >>>>>         - id: foo
> >>>>>           itemType: entity
> >>>>>           item:
> >>>>>             type: blah
> >>>>>
> >>>>> The general rule would be: if there is a version in the manifest
> >>>>> section, that will be used; if not then a top-level version number
> will
> >>>>> be used. If that is also missing, then fail.
> >>>>>
> >>>>> i.e. we would not accept:
> >>>>>
> >>>>>      # Fails - manifest has no version.
> >>>>>      brooklyn.catalog:
> >>>>>         manifest:
> >>>>>           symbolic_name: com.example.Foo
> >>>>>         items:
> >>>>>         - id: foo
> >>>>>           version: 1.0
> >>>>>           itemType: entity
> >>>>>           item:
> >>>>>             type: blah
> >>>>>
> >>>>> Thoughts?
> >>>>>
> >>>>> Aled
> >>>>>
> >>>>>
> >>>>> On 02/03/2017 10:02, Geoff Macartney wrote:
> >>>>>> I take Alex's point about the manifest being Java specific, and I
> >> agree
> >>>>>> therefore we shouldn't insist on it.
> >>>>>>
> >>>>>> +1 also to preferring explicit name/version in the catalog.bom
> rather
> >>>>> than
> >>>>>> as API params, I agree we do
> >>>>>> need to keep the version in source control.
> >>>>>>
> >>>>>> Question on your straw man, does the 'version' below
> >>>>>>
> >>>>>>       brooklyn.catalog:
> >>>>>>          manifest:
> >>>>>>            symbolic_name: com.example.Foo
> >>>>>>            version: 1.0.0-SNAPSHOT
> >>>>>>          items:
> >>>>>>
> >>>>>> _replace_ the 'version' in existing catalogs
> >>>>>>
> >>>>>> brooklyn.catalog:
> >>>>>>       version: "0.1.0-SNAPSHOT"
> >>>>>>
> >>>>>> or is it an independently variable value, i.e. the version of the
> >> bundle,
> >>>>>> separate from the version of catalog items that it happens to
> contain?
> >>>>>> (Sounds a bit disquieting, but I guess it's possible).
> >>>>>>
> >>>>>> e.g. could you have
> >>>>>>
> >>>>>> brooklyn.catalog:
> >>>>>>     version: 0.11.0-SNAPSHOT
> >>>>>>     manifest:
> >>>>>>       symbolic_name: com.example.Foo
> >>>>>>       version: 1.0.0-SNAPSHOT
> >>>>>>     items:
> >>>>>>     - id: foo
> >>>>>>       type: xyz
> >>>>>>
> >>>>>> which would be bundle com.example.Foo:1.0.0-SNAPSHOT, which happens
> to
> >>>>>> contain item foo:0.11.0-SNAPSHOT?
> >>>>>>
> >>>>>> If so, what would happen if you left out the second line above? What
> >>>>>> version of 'foo' would the catalog contain?
> >>>>>>
> >>>>>> Or does the version within the manifest _mean_ that it is not only
> the
> >>>>>> bundle version you are specifying, but the catalog item versions
> too?
> >> (I
> >>>>>> guess unless the item explicitly supplies its own version.)
> >>>>>>
> >>>>>> Geoff
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Wed, 1 Mar 2017 at 17:26 Aled Sage <al...@gmail.com> wrote:
> >>>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> Discussion of the REST api kicked off again in
> >>>>>>
> >>
> https://github.com/apache/brooklyn-server/pull/485#issuecomment-283280366:
> >>>>>>       Alex wrote:
> >>>>>>
> >>>>>>       Requiring a MANIFEST.MF makes the input strongly java-centric;
> >> I'd
> >>>>>>       like to appeal to people who write YAML blueprints with
> >> co-bundled
> >>>>>>       scripts and images. They'd wonder why they can't simply make
> and
> >>>>>>       upload a ZIP. They could probably be persuaded to supply a
> name
> >> and
> >>>>>>       optional version on a CLI or UI, and understand why it is
> needed
> >>>>>>       (hence supporting those args) but it would not be idiomatic to
> >>>>>>       anyone but a java programmer to create a META-INF/ dir with a
> >>>>>>       MANIFEST.MF and its syntax. Meanwhile it is very easy for us
> to
> >>>>>>       convert the ZIP to a JAR on the server. Feels like
> >> uncontroversial
> >>>>>>       good UX.
> >>>>>>
> >>>>>>       OTOH for a java programmer a MANIFEST.MF is natural, and
> they'd
> >> want
> >>>>>>       to drop the name/version args if they are in that file, and I
> see
> >> no
> >>>>>>       reason to forbid that pattern.
> >>>>>>
> >>>>>> I agree with Alex, that we should not require a
> META-INF/MANIFEST.MF.
> >> As
> >>>>>> for Geoff's suggestion that we could auto-generate a manifest in the
> >>>>>> `br` CLI, I'd prefer a more general solution that works for users of
> >> the
> >>>>>> REST api as well (i.e. doing it server-side).
> >>>>>>
> >>>>>> ---
> >>>>>> Svet suggested that the catalog.bom could give the symbolic name +
> >>>>>> version, via additional metadata in that file.
> >>>>>>
> >>>>>> What I really like about that is it's in version control, becuase
> it's
> >>>>>> in the file/repo. If alternatively the name/version are just passed
> as
> >>>>>> REST api parameters, then it's not in version control (and is more
> >>>>>> susceptible to typos etc).
> >>>>>>
> >>>>>> I'm not sure what we'd want this to look like. As a straw man:
> >>>>>>
> >>>>>>       brooklyn.catalog:
> >>>>>>          manifest:
> >>>>>>            symbolic_name: com.example.Foo
> >>>>>>            version: 1.0.0-SNAPSHOT
> >>>>>>          items:
> >>>>>>          - ...
> >>>>>>
> >>>>>> If the user built their own real OSGi bundle, then they wouldn't
> need
> >> to
> >>>>>> include this "manifest" section. If they did include that and it
> >>>>>> contradicted the OSGi bundle's manifest, then we'd fail-fast.
> >>>>>>
> >>>>>> (Note this reminds me of the (unrelated) metadata described in
> >>>>>> https://github.com/brooklyncentral/blueprint-repository- there is a
> >>>>>> "publish" block that can be added to the catalog.bom.)
> >>>>>>
> >>>>>> ---
> >>>>>> With Svet's suggestion, if there was no manifest section/file, then
> >>>>>> should we auto-generate something from the item(s) in the
> catalog.bom?
> >>>>>>
> >>>>>> I can see how that could easily work for a .bom file that has a
> single
> >>>>>> item (e.g. we use the catalog item's id + version, possibly with a
> >>>>>> special prefix in the symbolic name to avoid conflicts).
> >>>>>>
> >>>>>> However, if there are multiple items then it would get trickier.
> >>>>>>
> >>>>>> I'm inclined to say that for a minimal viable product (MVP) we can
> >>>>>> insist on the "manifest" section in the catalog.bom.
> >>>>>>
> >>>>>> Aled
> >>>>>>
> >>>>>>
> >>>>>> On 20/12/2016 16:34, Svetoslav Neykov wrote:
> >>>>>>>> Svet, if instead we tried to infer it from the catalog.bom, would
> we
> >>>>>> require some additional metadata within the .bom file? Or would we
> use
> >>>>> the
> >>>>>> catalog item's id + version? I'm not convinced by the latter - it
> >> would
> >>>>>> mean some .bom files would work and others wouldn't (e.g. if the
> .bom
> >> had
> >>>>>> multiple items with different versions). Better to support the
> >> explicit
> >>>>>> approach IMO.
> >>>>>>> I imagine it would be additional metadata. On the other hand I
> don't
> >> see
> >>>>>> a technical reason why we need an explicit symbolicName and version
> -
> >>>>> they
> >>>>>> can be auto-generated.
> >>>>>>> Svet.
> >>>>>>>
> >>>>>>>
> >>>>>>>> On 20.12.2016 г., at 17:50, Aled Sage <al...@gmail.com>
> wrote:
> >>>>>>>>
> >>>>>>>> Hi all,
> >>>>>>>>
> >>>>>>>> +1
> >>>>>>>>
> >>>>>>>> (D) sounds good. What version are you imagining the bundle would
> be,
> >> if
> >>>>>> one runs `br catalog add ~/my/project/ --name
> com.example.myproject`?
> >>>>>>>> ---
> >>>>>>>> I like the idea of uploading a plain zip (rather than only
> >> supporting
> >>>>>> OSGi bundles) - that makes it simpler for non-java folk. The use of
> >> OSGi
> >>>>>> becomes a (hidden) implementation detail to many users.
> >>>>>>>> ---
> >>>>>>>> If auto-generating the manifest, I think we need the user to be
> >>>>> explicit
> >>>>>> about symbolic name and version. Having these supplied in the REST
> api
> >>>>> call
> >>>>>> (as Alex suggests) would achieve that.
> >>>>>>>> Svet, if instead we tried to infer it from the catalog.bom, would
> we
> >>>>>> require some additional metadata within the .bom file? Or would we
> use
> >>>>> the
> >>>>>> catalog item's id + version? I'm not convinced by the latter - it
> >> would
> >>>>>> mean some .bom files would work and others wouldn't (e.g. if the
> .bom
> >> had
> >>>>>> multiple items with different versions). Better to support the
> >> explicit
> >>>>>> approach IMO.
> >>>>>>>> ---
> >>>>>>>> For E ("have a mechanism whereby deployed entities based on an
> >> affected
> >>>>>> blueprint are optionally migrated to the new code"), that feels
> like a
> >>>>>> separate discussion. It could equally apply to a pure YAML .bom file
> >> that
> >>>>>> has been added to the catalog.
> >>>>>>>> I suggest we discuss that in a separate email thread.
> >>>>>>>>
> >>>>>>>> ---
> >>>>>>>> For (G), it's an interesting suggestion from Svet to make use of
> >> Karaf
> >>>>>> Cellar for HA nodes. I'm hesitant (e.g. if restarting a standalone
> >>>>> Brooklyn
> >>>>>> node whose VM has died, then it adds big additional requirements for
> >> what
> >>>>>> constitutes the "persisted state"). On the other hand, it's good to
> >> use
> >>>>>> well-established technologies rather than re-inventing things!
> >>>>>>>> An alternative ("pure brooklyn") approach could be to write the
> >> bundle
> >>>>>> to persisted state; on rebind, we'd install + activate those
> bundles.
> >>>>>>>> ---
> >>>>>>>> For "catalogGroupId", I agree with Svet that in the initial
> use-case
> >>>>>> this can be an implementation detail.
> >>>>>>>> It could be set as the bundle's symbolic name + version:
> everything
> >>>>> from
> >>>>>> the bundle should be deleted at once, along with the bundle.
> >>>>>>>> Longer term, I can see how exposing "catalogGroupId" to the user
> >> could
> >>>>>> support more use-cases (e.g. for several catalog items from
> different
> >>>>>> bundles to work together). I don't think we should try to support
> that
> >>>>> yet.
> >>>>>>>> Aled
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 19/12/2016 17:19, Geoff Macartney wrote:
> >>>>>>>>> hi Alex,
> >>>>>>>>>
> >>>>>>>>> this looks like a good feature to have, I shall look at the PR as
> >> soon
> >>>>>> as I
> >>>>>>>>> can.
> >>>>>>>>>
> >>>>>>>>> The catalog.bom scanner feature was initially enabled by default,
> >> but
> >>>>> we
> >>>>>>>>> had to
> >>>>>>>>> disable it because it turned out not to work properly with
> rebind.
> >> I
> >>>>>> don't
> >>>>>>>>> think
> >>>>>>>>> it should be a lot of work to fix that but it hasn't been
> something
> >>>>>> we've
> >>>>>>>>> got round
> >>>>>>>>> to yet.  This would be a great opportunity to look back at that.
> >>>>>>>>>
> >>>>>>>>> Some random thoughts:
> >>>>>>>>>
> >>>>>>>>> re (C), if we are going to treat the zips as bundles, my gut feel
> >> is
> >>>>>> that
> >>>>>>>>> we
> >>>>>>>>> should insist on a manifest and get the metadata from it.  It
> >> doesn't
> >>>>>> feel
> >>>>>>>>> to me
> >>>>>>>>> like it makes much sense to allow a zip file without a
> MANIFEST.MF
> >> but
> >>>>>>>>> convey
> >>>>>>>>> the intended bundle metadata to Brooklyn via HTTP headers.  And
> >> rather
> >>>>>> than
> >>>>>>>>> infer bundle metadata I think it's better to ask users to be
> >> explicit
> >>>>>> about
> >>>>>>>>> what
> >>>>>>>>> their intentions are.  To make users lives easier, we could
> >>>>>>>>> add a command to br to generate the manifest (locally) with
> correct
> >>>>>> syntax,
> >>>>>>>>> so that the manifest is in the right place, rather than have br
> add
> >>>>> the
> >>>>>> data
> >>>>>>>>> to the "upload" request headers.
> >>>>>>>>>
> >>>>>>>>> re. (D) will be glad to have a look at it
> >>>>>>>>>
> >>>>>>>>> re. (E) it would certainly need to be optional - maybe keep it as
> >> an
> >>>>>>>>> explicit
> >>>>>>>>> separate command ('upgrade'?)
> >>>>>>>>>
> >>>>>>>>> (F) it does seem like a lot of work but might be nice for users
> who
> >>>>> are
> >>>>>> not
> >>>>>>>>> keen on command lines.
> >>>>>>>>>
> >>>>>>>>> G - I:  we'll definitely need to pay close attention to
> persistence
> >>>>> and
> >>>>>>>>> rebind;
> >>>>>>>>> I wonder also about HA operation, are there any additional
> >>>>> implications?
> >>>>>>>>> (J) I think it would be good to treat all the files from a jar,
> >> sorry
> >>>>>>>>> bundle,
> >>>>>>>>> as an atomic group - cleaner that way perhaps than allowing
> >>>>>> delete/update
> >>>>>>>>> of
> >>>>>>>>> individual entries from a bundle on a piecemeal basis.  Rest
> >> support
> >>>>> on
> >>>>>>>>> delete
> >>>>>>>>> catalog could warn about related catalog entries being deleted
> and
> >> ask
> >>>>>> for
> >>>>>>>>> a "--force" param to confirm.
> >>>>>>>>>
> >>>>>>>>> Geoff
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Fri, 16 Dec 2016 at 15:24 Svetoslav Neykov <
> >>>>>>>>> svetoslav.neykov@cloudsoftcorp.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> +1
> >>>>>>>>>>
> >>>>>>>>>> Some thoughts:
> >>>>>>>>>>      * (A) add a utility class BundleMaker
> >>>>>>>>>>        Sounds very similar to
> >>>>>>>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles <
> >>>>>>>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles>
> >>>>>>>>>>        Looking at the code it's much more focused on zip files
> so I
> >>>>> guess
> >>>>>>>>>> there's no much overlap, but worth keeping in mind
> >>>>>>>>>>      * (C) accept bundle symbolic name and version
> >>>>>>>>>>        Why require them at all? Could infer them from the
> >> catalog.bom
> >>>>> in
> >>>>>> some
> >>>>>>>>>> way - maybe require those properties to be in there. If not
> >> present
> >>>>> are
> >>>>>>>>>> they really needed?
> >>>>>>>>>>      * (G) Bundles installed via this mechanism are not
> persisted
> >>>>>> currently &
> >>>>>>>>>> (I) We persist the individual catalog items as YAML, so we end
> up
> >>>>> with
> >>>>>> two
> >>>>>>>>>> records
> >>>>>>>>>>        Suggest marking the catalog items coming from bundles as
> >>>>>>>>>> non-persistable. Then try to share the bundles between HA nodes.
> >>>>> (Karaf
> >>>>>>>>>> Cellar?)
> >>>>>>>>>>      * (J) Introduce a catalogGroupId field on catalog items;
> >>>>>>>>>>        Agree this could be useful and I like the idea of
> deleting
> >> the
> >>>>>> bundle
> >>>>>>>>>> altogether with the catalog items. From user's perspective I
> don't
> >>>>> see
> >>>>>> the
> >>>>>>>>>> need for an extra field (i.e. it's an implementation detail).
> >>>>>>>>>>
> >>>>>>>>>> Svet.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> On 16.12.2016 г., at 12:50, Alex Heneveld <
> >>>>>>>>>> alex.heneveld@cloudsoftcorp.com> wrote:
> >>>>>>>>>>> Hi Brooklyners-
> >>>>>>>>>>>
> >>>>>>>>>>> In the code we currently have two routes for users to install
> new
> >>>>>>>>>>> blueprints:
> >>>>>>>>>>>
> >>>>>>>>>>> (1) upload a catalog YAML file to /v1/catalog
> >>>>>>>>>>>
> >>>>>>>>>>> (2) install a bundle with catalog.bom in the root
> >>>>>>>>>>>
> >>>>>>>>>>> The feature (2) is disabled by default, but I'd like to move
> >> towards
> >>>>>>>>>>> enabling it.  This will make it easier to create nicely
> >> structured
> >>>>> BOM
> >>>>>>>>>>> files because scripts etc can be taken out of the BOM, stored
> as
> >>>>>> files in
> >>>>>>>>>>> the same bundle.  (Because URLs of the form
> >>>>>>>>>>> `classpath://scripts/install.sh`  use the bundle's classpath to
> >>>>>> resolve.)
> >>>>>>>>>>> As a first step in #485 [1] I do a few things:
> >>>>>>>>>>>
> >>>>>>>>>>> (A) add a utility class BundleMaker that lets us create and
> >> modify
> >>>>>>>>>>> bundles/zips, to make it easier to do things we might want to
> >> with
> >>>>>>>>>> bundles,
> >>>>>>>>>>> especially for testing
> >>>>>>>>>>>
> >>>>>>>>>>> (B) add an endpoint to the REST API which allows uploading a
> >> bundle
> >>>>>> ZIP
> >>>>>>>>>>> (C) accept bundle symbolic name and version in that REST API to
> >>>>>>>>>> facilitate
> >>>>>>>>>>> uploading non-bundle ZIPs where the OSGi MANIFEST.MF is
> >>>>> automatically
> >>>>>>>>>>> generated
> >>>>>>>>>>>
> >>>>>>>>>>> With this PR, if you have a directory on your local file system
> >> with
> >>>>>>>>>>> scripts and config files, and a BOM which refers to them, you
> can
> >>>>> just
> >>>>>>>>>> ZIP
> >>>>>>>>>>> that up an upload it, specifying the bundle name so that a YAML
> >>>>>> blueprint
> >>>>>>>>>>> author never needs to touch any java-isms.
> >>>>>>>>>>>
> >>>>>>>>>>> Where I see this going is a development workflow where a user
> can
> >>>>> edit
> >>>>>>>>>>> files locally and upload the ZIP to have that installed, and if
> >> they
> >>>>>> make
> >>>>>>>>>>> changes locally they can POST it again to have catalog items
> >> updated
> >>>>>>>>>>> (because default version is a SNAPSHOT).  We could also:
> >>>>>>>>>>>
> >>>>>>>>>>> (D) have `br catalog add ~/my/project/ --name my.project`
> create
> >> a
> >>>>> ZIP
> >>>>>>>>>> and
> >>>>>>>>>>> POST it, with bundle name metadata, so essentially the user's
> >>>>> process
> >>>>>> is
> >>>>>>>>>>> just to run that whenever they make a change
> >>>>>>>>>>>
> >>>>>>>>>>> (E) have a mechanism whereby deployed entities based on an
> >> affected
> >>>>>>>>>>> blueprint are optionally migrated to the new code, so if you've
> >>>>>> changed
> >>>>>>>>>> an
> >>>>>>>>>>> enricher the changes are picked up, or if say a launch.sh
> script
> >> has
> >>>>>>>>>>> changed, a restart will run the new code
> >>>>>>>>>>>
> >>>>>>>>>>> The above are fairly straightforward programmatically (although
> >> good
> >>>>>> user
> >>>>>>>>>>> interaction with (E) needs some thought).  So I think we can
> >> pretty
> >>>>>>>>>> quickly
> >>>>>>>>>>> get to a much smoother dev workflow.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> That's the highlight of this message.  You can jump to the end,
> >>>>> unless
> >>>>>>>>>>> you're interested in some important but low level details...
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> I'm also tempted by:
> >>>>>>>>>>>
> >>>>>>>>>>> (F) Integration with web-based IDE and/or Brooklyn reading and
> >>>>> writing
> >>>>>>>>>>> straight from GitHub -- but this seems like a lot of work and
> I'm
> >>>>> not
> >>>>>>>>>>> convinced it's much better than (D) workflow-wise
> >>>>>>>>>>>
> >>>>>>>>>>> Before we can change (2) to be the default, or start widely
> using
> >>>>> the
> >>>>>>>>>> POST
> >>>>>>>>>>> a ZIP feature, we need to sort out some issues to do with
> >>>>> persistence
> >>>>>> and
> >>>>>>>>>>> reloading:
> >>>>>>>>>>>
> >>>>>>>>>>> (G) Bundles installed via this mechanism are not persisted
> >>>>> currently,
> >>>>>> so
> >>>>>>>>>> if
> >>>>>>>>>>> you move to a different Brooklyn using the same backing store,
> >>>>> you'll
> >>>>>>>>>> lose
> >>>>>>>>>>> those bundles
> >>>>>>>>>>>
> >>>>>>>>>>> (H) On rebind, bundles aren't always activated when needed,
> >> meaning
> >>>>>> items
> >>>>>>>>>>> can't be loaded
> >>>>>>>>>>>
> >>>>>>>>>>> (I) We persist the individual catalog items as YAML, so we end
> up
> >>>>> with
> >>>>>>>>>> two
> >>>>>>>>>>> records — the YAML from the catalog.bom in the bundle, and the
> >> YAML
> >>>>>>>>>>> persisted for the item.  This isn't a problem per se, but
> >> something
> >>>>> to
> >>>>>>>>>>> think about, and some sometimes surprising behaviour.  In
> >> particular
> >>>>>> if
> >>>>>>>>>> you
> >>>>>>>>>>> delete the persisted YAML, the bundle is still there so the
> item
> >> is
> >>>>> no
> >>>>>>>>>>> longer deleted after a full rebind.
> >>>>>>>>>>>
> >>>>>>>>>>> One idea which might be useful is:
> >>>>>>>>>>>
> >>>>>>>>>>> (J) Introduce a catalogGroupId field on catalog items; this
> will
> >> do
> >>>>>> two
> >>>>>>>>>>> things:  if you try to delete an item with such a record,
> you'll
> >> be
> >>>>>>>>>>> encouraged to delete all such items (maybe disallowed to delete
> >> an
> >>>>>>>>>>> individual one), with the effect of deleting the bundle if it
> >> comes
> >>>>>> from
> >>>>>>>>>> a
> >>>>>>>>>>> bundle; and when resolving types we search first for items with
> >> the
> >>>>>> same
> >>>>>>>>>>> catalogGroupId (so that e.g. if I install MyCluster:1.0 and
> >>>>>> MyNode:1.0 in
> >>>>>>>>>>> the same group, the former can refer simply to "MyNode" but if
> I
> >>>>>> install
> >>>>>>>>>> a
> >>>>>>>>>>> 2.0 version of that group, the 1.0 cluster still loads the 1.0
> >> node
> >>>>> --
> >>>>>>>>>> this
> >>>>>>>>>>> has bitten people i the past)
> >>>>>>>>>>>
> >>>>>>>>>>> There is a related Brooklyn upgrade problem worth mentioning,
> >> which
> >>>>>> the
> >>>>>>>>>>> above might help with, where:
> >>>>>>>>>>>
> >>>>>>>>>>> (K) If I migrate from Brooklyn 10 to 11 when it comes out, I'll
> >> no
> >>>>>> longer
> >>>>>>>>>>> have certain entities that were at v10, since we don't include
> >>>>> those;
> >>>>>> an
> >>>>>>>>>>> upgrade could include rules that certain groupIds need to be
> >>>>> updated,
> >>>>>> or
> >>>>>>>>>> it
> >>>>>>>>>>> can search and attempt to automatically apply the updates
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Quite a lot here and we don't need to solve it but I wanted to:
> >>>>>>>>>>>
> >>>>>>>>>>> * Share the current thinking
> >>>>>>>>>>>
> >>>>>>>>>>> * Get opinions on the general dev workflow suggested by (D)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for feedback -- and if we like it help with (D) would be
> >>>>>>>>>> appreciated!
> >>>>>>>>>>> Best
> >>>>>>>>>>> Alex
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> [1] . https://github.com/apache/brooklyn-server/pull/485
> >> --
> >>
> >> Thomas Bouron • Software Engineer @ Cloudsoft Corporation •
> >> http://www.cloudsoftcorp.com/
> >> Github: https://github.com/tbouron
> >> Twitter: https://twitter.com/eltibouron
> >>
> >> --
> >>
> >> Thomas Bouron • Software Engineer @ Cloudsoft Corporation •
> >> http://www.cloudsoftcorp.com/
> >> Github: https://github.com/tbouron
> >> Twitter: https://twitter.com/eltibouron
> >>
>
> --
>
> Thomas Bouron • Software Engineer @ Cloudsoft Corporation •
> http://www.cloudsoftcorp.com/
> Github: https://github.com/tbouron
> Twitter: https://twitter.com/eltibouron
>
-- 

Thomas Bouron • Software Engineer @ Cloudsoft Corporation •
http://www.cloudsoftcorp.com/
Github: https://github.com/tbouron
Twitter: https://twitter.com/eltibouron

Re: Uploading ZIPs for a better dev workflow

[Thomas Bouron <th...@cloudsoftcorp.com>]

Hi Aled, all.

Sounds good to me. Once the PR is merged, I'll push on another one to fix
the other issues I found in the past 2 days.

I also agree with you on 2 and 3, I'll address that.

Best.

On Thu, 23 Mar 2017 at 11:02 Aled Sage <al...@gmail.com> wrote:

> Hi Thomas, all,
>
> I think "bundle: ..." is actually a much better name to use in the
> catalog.bom than "symbolicName: ...". The latter makes sense to those
> who know OSGi, but otherwise doesn't suggest anything about "bundling".
>
> ---
>
> I think we're very close to merging [1] - further improvements can
> continue incrementally in subsequent PRs, rather than all of this having
> to be done in [1].
>
> ---
>
> There's a change compared to the mailing list discussion, I think [2,3].
> Alex has implemented it so that if "symbolicName" and "id" are used at
> the top-level of the catalog.bom, then they *must match*. This seems
> wrong to me. It's conflating the name spaces of OSGi bundles and catalog
> items. I suggest we revert back to (at least how I interpretted) the
> previous mailing list discussion: if "bundle: " is present, then that is
> used; if "bundle" is absent, then we use "id"; if both are absent, then
> we fail.
>
> Aled
>
> p.s. As an aside, we should deprecate (and soon delete) support for
> "symbolicName" in a catalog.bom - one should use id/version. Nowhere in
> our docs to we say symbolicName. It's just yet another example of us
> supporting multiple ways of doing the same thing, where it's not at all
> clear they mean the same thing, and risking confusing users.
>
> [1] https://github.com/apache/brooklyn-server/pull/485
> [2]
> https://github.com/apache/brooklyn-server/pull/485#discussion_r107144368
> [3]
> https://github.com/apache/brooklyn-server/pull/485#issuecomment-288052012
>
>
> On 23/03/2017 10:23, Geoff Macartney wrote:
> > some further thoughts:
> >
> > re. 1 I agree we should discriminate between the "symbolicName" of the
> > bundle and use of the term "symbolicName" in the definition of catalog
> ids.
> > (I'm not sure if just putting it into a "manifest" subsection will keep
> it
> > separate, because of the use of "getFirstAs" at
> >
> https://github.com/apache/brooklyn-server/blob/master/core/src/main/java/org/apache/brooklyn/core/catalog/internal/BasicBrooklynCatalog.java#L542
> > ).
> >
> > re. use of "CatalogPopulator" I think we shall want to factor apart
> > CatalogBomScanner/CatalogPopulator into separate classes, -
> > - CatalogBomScanner to be the management context service listener
> > - CatalogPopulator becomes:
> > -- CatalogBundleTracker to do the BundleTracker stuff
> > -- CatalogBundleLoader - separate out the handling of the loading of the
> > "catalog.bom" from a given bundle
> >
> > and then 485 can use CatalogBundleLoader.  We'll still need to think
> about
> > the delete side of things, and handling errors, as Thomas says.
> >
> > re. the Swagger issue, an alternative would be that we could just drop
> > "createFromYaml" and "createFromArchive" and let "createPoly" continue to
> > do as it does, handling either case; maybe best renamed
> "createFromUpload".
> >
> > Geoff
> >
> > On Thu, 23 Mar 2017 at 09:40 Thomas Bouron <
> thomas.bouron@cloudsoftcorp.com>
> > wrote:
> >
> >> Hi all.
> >>
> >> So I spent time to review and test [1] and here are the few issues I
> bumped
> >> into:
> >>
> >>     1. the use of `symbolicName` for the bundle clashes Brooklyn: the
> >>     `BasicBrooklynCatalog` watches this key to define catalog items IDs
> [2]
> >>     2. because of 1. the bundle that I'm trying to install fails. But
> upon
> >>     failure, the bundle is not uninstalled so any subsequent bundle
> addition
> >>     fail with the first error
> >>     3. we now have multiple API definitions for the same endpoint (i.e.
> >>     /v1/catalog for yaml and ZIP/JAR) While this is perfectly valid,
> Swagger
> >>     does not support it [3]. It means that the Swagger UI displays only
> the
> >>     latest API definition registered, in this case, that's the one to
> upload
> >>     ZIP/JAR.
> >>
> >> For 1. I can see 2 solutions: using a special section `manifest` where
> we
> >> put the `symbolicName` and `version`, or changing the field name to
> >> `bundle` (I prefer option 2)
> >> For 2. we need to add some logic to uninstall the bundle on failure into
> >> the `CatalogPopulator`, as per as Geoff comment[4]
> >> For 3. even the new Swagger spec (called now OpenAPI) does not support
> >> multiple definitions on the same endpoint [5]. I'm not sure if we can
> leave
> >> with that. One solution would be to split the endpoint to create
> specific
> >> one, based on the type of upload.
> >>
> >> [1] https://github.com/apache/brooklyn-server/pull/485
> >> [2]
> >>
> >>
> https://github.com/apache/brooklyn-server/blob/master/core/src/main/java/org/apache/brooklyn/core/catalog/internal/BasicBrooklynCatalog.java#L542
> >> [3] https://github.com/swagger-api/swagger-core/issues/935
> >> [4]
> >>
> https://github.com/apache/brooklyn-server/pull/485#discussion_r107482279
> >> [5] https://github.com/OAI/OpenAPI-Specification/issues/182
> >>
> >> On Mon, 20 Mar 2017 at 15:16 Thomas Bouron <
> >> thomas.bouron@cloudsoftcorp.com>
> >> wrote:
> >>
> >> Hi Brooklyners.
> >>
> >> Uploading ZIPs is really something I'm excited to see coming to life so
> I
> >> would like to help to push this forward.
> >>
> >> I've re-read this email thread and it sounds like we have a consensus on
> >> the "one true way" approach, i.e. always requiring the `symbolicName`
> and
> >> `version` when uploading ZIP/JAR (that includes Svet in his previous
> >> email).
> >>
> >> One thing not explicitly settled is the format of bom for those 2
> fields:
> >> Alex prefers to put these at the root, Aled in a `manifest` section. I
> >> don't have any strong opinion on this: I was leaning toward Aled's
> >> suggestion but the argument that says this applies to the whole bundle
> make
> >> sense so I now think putting this at the root is probably better. I can
> see
> >> that Alex already updated his PR[1] to this structure. What the people
> >> think about this?
> >>
> >> I'm going to review [1] but are we happy with this consensus? One thing
> >> that will need to be addressed is the persistence/rebind issues but this
> >> can be tackled in following PRs.
> >>
> >> Best.
> >>
> >> [1] https://github.com/apache/brooklyn-server/pull/485
> >>
> >> On Tue, 7 Mar 2017 at 16:18 Svetoslav Neykov <
> >> svetoslav.neykov@cloudsoftcorp.com> wrote:
> >>
> >> Having quickly scanned the thread I still don't see a reason why the
> >> requirement to have a consistent naming for the bundle itself which is
> not
> >> user visible? Could instead have behaviour similar to the machinNamer
> here.
> >>
> >> Slight preference for optional symbolicName when it's a jar file.
> Though I
> >> really like Geoff's idea of "one true way". The workflow to create a jar
> >> file is very different from creating a zip file (either from a folder or
> >> user supplied) so don't think they conflict.
> >>
> >> Svet.
> >>
> >>
> >>> On 7.03.2017 г., at 17:47, Alex Heneveld <
> >> alex.heneveld@cloudsoftcorp.com>
> >> wrote:
> >>>
> >>> Considering the points made I'm pretty sure it's the case that
> requiring
> >> a symbolic name and version in the BOM presents little if any
> unnecessary
> >> burden, in which case I'm persuaded of the "one true way" philosophy.  I
> >> really like it.  :)
> >>> Can anyone think of a case where it would be a burden?  My thinking is
> >> that:
> >>> * If you're uploading a ZIP of YAML and scripts and icons, you have a
> >> catalog.bom and that's a useful place to require the symname and version
> >>> * If your ZIP/JAR also has a MANIFEST.MF the symname and version will
> be
> >> present in it also; the user is duplicating that info (this is the
> "little"
> >> unnecssary burden) but it's tiny and the benefit of one true way
> dominates;
> >> we do require that they match
> >>> * If your ZIP/JAR doesn't have a BOM there's no point in uploading it
> to
> >> the catalog endpoint; either it's a JAR but this isn't meant as an
> "install
> >> arbitrary bundles" endpoint, or a non-JAR in which case what are we
> >> supposed to do with it once you upload it?
> >>> It keeps the contract simple, and the CLI very simple.   `br push
> >> /path/to/blueprint/x/`
> >>> (But Geoff as a side point I think the CLI should _not_ attempt
> >> validation of the input/version -- the server will do that anyway and no
> >> benefit in doing it twice, but more code to maintain.)
> >>> I'm further persuaded given it is in a file, we should require a
> >> version.  I was concerned about needing to supply the version every
> time on
> >> the CLI but this has gone away.
> >>> I see no reason for a separate "manifest" section.  Removing this
> >> eliminates the prospect of Aled's pathological/"perverted" example.  I
> >> favour:
> >>>     brooklyn.catalog:
> >>>        symbolicName: com.example.Foo
> >>>        version: 1.0
> >>>        items:
> >>>        - id: foo
> >>>          version: 3.0   # optional version override
> >>>          itemType: entity
> >>>          item:
> >>>            type: blah
> >>>            ...
> >>>
> >>> That is, the fields symbolic-name and version are _required_ under the
> >> `brooklyn.catalog` when POSTing a ZIP to the catalog endpoint, and are
> used
> >> to define the bundle.  The version (and icon, etc) are then inherited by
> >> default by items defined therein.  If the ZIP is a JAR which contains a
> >> MANIFEST.MF, the bundle information (symbolic name and version) must
> match
> >> exactly.
> >>> Sound right?
> >>>
> >>> Best
> >>> Alex
> >>>
> >>>
> >>> On 03/03/2017 09:59, Geoff Macartney wrote:
> >>>> It also means clients ('br' at least) can easily validate the upload
> >> even
> >>>> before execution by checking that all the manifest details are
> present.
> >>>>
> >>>> On Fri, 3 Mar 2017 at 09:57 Geoff Macartney <
> >>>> geoff.macartney@cloudsoftcorp.com> wrote:
> >>>>
> >>>>> Just on the last point of "We could also support the example below",
> >> I'd
> >>>>> say let's not even be that flexible - my feeling is that the more
> >> flexible
> >>>>> we are, the more confusing it is, and the harder to get right.  If
> >> we're
> >>>>> going to add this capability let's just have one way to do it; if you
> >> are
> >>>>> going to do a ZIP upload, you MUST have a catalog.bom, which MUST
> >> contain a
> >>>>> manifest section, which MUST have both a symbolic name and a version.
> >> That
> >>>>> way everything's explicit and very clear, every time.
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Thu, 2 Mar 2017 at 11:45 Aled Sage <al...@gmail.com> wrote:
> >>>>>
> >>>>> Geoff, all,
> >>>>>
> >>>>> I was imagining the manifest version (in the catalog.bom) to be
> >> separate
> >>>>> from the item version. The reason is that we support multiple items
> in
> >>>>> the bom that can be independently versioned.
> >>>>>
> >>>>> Somone perverted could write:
> >>>>>
> >>>>>      brooklyn.catalog:
> >>>>>         version: 1.0
> >>>>>         manifest:
> >>>>>           symbolic_name: com.example.Foo
> >>>>>           version: 2.0
> >>>>>         items:
> >>>>>         - id: foo
> >>>>>           version: 3.0
> >>>>>           itemType: entity
> >>>>>           item:
> >>>>>             type: blah
> >>>>>         - id: bar
> >>>>>           version: 4.0
> >>>>>           itemType: entity
> >>>>>           item:
> >>>>>          type: blah.blah
> >>>>>
> >>>>> Here, the "1.0" version is not used by anything; the auto-generated
> >> OSGi
> >>>>> bundle would be version 2.0; catalog item foo would be 3.0; and
> catalog
> >>>>> item bar would be 4.0.
> >>>>>
> >>>>> If we were starting from scratch, I'd be tempted to lock things down
> a
> >>>>> lot more for what we actually support. For now, it feels like asking
> >> for
> >>>>> an explicit version is a reasonable thing to do.
> >>>>>
> >>>>> ---
> >>>>> We could also support the example below, where "1.0" is used by both
> >> the
> >>>>> manifest and the item:
> >>>>>
> >>>>>      brooklyn.catalog:
> >>>>>         version: 1.0
> >>>>>         manifest:
> >>>>>           symbolic_name: com.example.Foo
> >>>>>         items:
> >>>>>         - id: foo
> >>>>>           itemType: entity
> >>>>>           item:
> >>>>>             type: blah
> >>>>>
> >>>>> The general rule would be: if there is a version in the manifest
> >>>>> section, that will be used; if not then a top-level version number
> will
> >>>>> be used. If that is also missing, then fail.
> >>>>>
> >>>>> i.e. we would not accept:
> >>>>>
> >>>>>      # Fails - manifest has no version.
> >>>>>      brooklyn.catalog:
> >>>>>         manifest:
> >>>>>           symbolic_name: com.example.Foo
> >>>>>         items:
> >>>>>         - id: foo
> >>>>>           version: 1.0
> >>>>>           itemType: entity
> >>>>>           item:
> >>>>>             type: blah
> >>>>>
> >>>>> Thoughts?
> >>>>>
> >>>>> Aled
> >>>>>
> >>>>>
> >>>>> On 02/03/2017 10:02, Geoff Macartney wrote:
> >>>>>> I take Alex's point about the manifest being Java specific, and I
> >> agree
> >>>>>> therefore we shouldn't insist on it.
> >>>>>>
> >>>>>> +1 also to preferring explicit name/version in the catalog.bom
> rather
> >>>>> than
> >>>>>> as API params, I agree we do
> >>>>>> need to keep the version in source control.
> >>>>>>
> >>>>>> Question on your straw man, does the 'version' below
> >>>>>>
> >>>>>>       brooklyn.catalog:
> >>>>>>          manifest:
> >>>>>>            symbolic_name: com.example.Foo
> >>>>>>            version: 1.0.0-SNAPSHOT
> >>>>>>          items:
> >>>>>>
> >>>>>> _replace_ the 'version' in existing catalogs
> >>>>>>
> >>>>>> brooklyn.catalog:
> >>>>>>       version: "0.1.0-SNAPSHOT"
> >>>>>>
> >>>>>> or is it an independently variable value, i.e. the version of the
> >> bundle,
> >>>>>> separate from the version of catalog items that it happens to
> contain?
> >>>>>> (Sounds a bit disquieting, but I guess it's possible).
> >>>>>>
> >>>>>> e.g. could you have
> >>>>>>
> >>>>>> brooklyn.catalog:
> >>>>>>     version: 0.11.0-SNAPSHOT
> >>>>>>     manifest:
> >>>>>>       symbolic_name: com.example.Foo
> >>>>>>       version: 1.0.0-SNAPSHOT
> >>>>>>     items:
> >>>>>>     - id: foo
> >>>>>>       type: xyz
> >>>>>>
> >>>>>> which would be bundle com.example.Foo:1.0.0-SNAPSHOT, which happens
> to
> >>>>>> contain item foo:0.11.0-SNAPSHOT?
> >>>>>>
> >>>>>> If so, what would happen if you left out the second line above? What
> >>>>>> version of 'foo' would the catalog contain?
> >>>>>>
> >>>>>> Or does the version within the manifest _mean_ that it is not only
> the
> >>>>>> bundle version you are specifying, but the catalog item versions
> too?
> >> (I
> >>>>>> guess unless the item explicitly supplies its own version.)
> >>>>>>
> >>>>>> Geoff
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Wed, 1 Mar 2017 at 17:26 Aled Sage <al...@gmail.com> wrote:
> >>>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> Discussion of the REST api kicked off again in
> >>>>>>
> >>
> https://github.com/apache/brooklyn-server/pull/485#issuecomment-283280366:
> >>>>>>       Alex wrote:
> >>>>>>
> >>>>>>       Requiring a MANIFEST.MF makes the input strongly java-centric;
> >> I'd
> >>>>>>       like to appeal to people who write YAML blueprints with
> >> co-bundled
> >>>>>>       scripts and images. They'd wonder why they can't simply make
> and
> >>>>>>       upload a ZIP. They could probably be persuaded to supply a
> name
> >> and
> >>>>>>       optional version on a CLI or UI, and understand why it is
> needed
> >>>>>>       (hence supporting those args) but it would not be idiomatic to
> >>>>>>       anyone but a java programmer to create a META-INF/ dir with a
> >>>>>>       MANIFEST.MF and its syntax. Meanwhile it is very easy for us
> to
> >>>>>>       convert the ZIP to a JAR on the server. Feels like
> >> uncontroversial
> >>>>>>       good UX.
> >>>>>>
> >>>>>>       OTOH for a java programmer a MANIFEST.MF is natural, and
> they'd
> >> want
> >>>>>>       to drop the name/version args if they are in that file, and I
> see
> >> no
> >>>>>>       reason to forbid that pattern.
> >>>>>>
> >>>>>> I agree with Alex, that we should not require a
> META-INF/MANIFEST.MF.
> >> As
> >>>>>> for Geoff's suggestion that we could auto-generate a manifest in the
> >>>>>> `br` CLI, I'd prefer a more general solution that works for users of
> >> the
> >>>>>> REST api as well (i.e. doing it server-side).
> >>>>>>
> >>>>>> ---
> >>>>>> Svet suggested that the catalog.bom could give the symbolic name +
> >>>>>> version, via additional metadata in that file.
> >>>>>>
> >>>>>> What I really like about that is it's in version control, becuase
> it's
> >>>>>> in the file/repo. If alternatively the name/version are just passed
> as
> >>>>>> REST api parameters, then it's not in version control (and is more
> >>>>>> susceptible to typos etc).
> >>>>>>
> >>>>>> I'm not sure what we'd want this to look like. As a straw man:
> >>>>>>
> >>>>>>       brooklyn.catalog:
> >>>>>>          manifest:
> >>>>>>            symbolic_name: com.example.Foo
> >>>>>>            version: 1.0.0-SNAPSHOT
> >>>>>>          items:
> >>>>>>          - ...
> >>>>>>
> >>>>>> If the user built their own real OSGi bundle, then they wouldn't
> need
> >> to
> >>>>>> include this "manifest" section. If they did include that and it
> >>>>>> contradicted the OSGi bundle's manifest, then we'd fail-fast.
> >>>>>>
> >>>>>> (Note this reminds me of the (unrelated) metadata described in
> >>>>>> https://github.com/brooklyncentral/blueprint-repository- there is a
> >>>>>> "publish" block that can be added to the catalog.bom.)
> >>>>>>
> >>>>>> ---
> >>>>>> With Svet's suggestion, if there was no manifest section/file, then
> >>>>>> should we auto-generate something from the item(s) in the
> catalog.bom?
> >>>>>>
> >>>>>> I can see how that could easily work for a .bom file that has a
> single
> >>>>>> item (e.g. we use the catalog item's id + version, possibly with a
> >>>>>> special prefix in the symbolic name to avoid conflicts).
> >>>>>>
> >>>>>> However, if there are multiple items then it would get trickier.
> >>>>>>
> >>>>>> I'm inclined to say that for a minimal viable product (MVP) we can
> >>>>>> insist on the "manifest" section in the catalog.bom.
> >>>>>>
> >>>>>> Aled
> >>>>>>
> >>>>>>
> >>>>>> On 20/12/2016 16:34, Svetoslav Neykov wrote:
> >>>>>>>> Svet, if instead we tried to infer it from the catalog.bom, would
> we
> >>>>>> require some additional metadata within the .bom file? Or would we
> use
> >>>>> the
> >>>>>> catalog item's id + version? I'm not convinced by the latter - it
> >> would
> >>>>>> mean some .bom files would work and others wouldn't (e.g. if the
> .bom
> >> had
> >>>>>> multiple items with different versions). Better to support the
> >> explicit
> >>>>>> approach IMO.
> >>>>>>> I imagine it would be additional metadata. On the other hand I
> don't
> >> see
> >>>>>> a technical reason why we need an explicit symbolicName and version
> -
> >>>>> they
> >>>>>> can be auto-generated.
> >>>>>>> Svet.
> >>>>>>>
> >>>>>>>
> >>>>>>>> On 20.12.2016 г., at 17:50, Aled Sage <al...@gmail.com>
> wrote:
> >>>>>>>>
> >>>>>>>> Hi all,
> >>>>>>>>
> >>>>>>>> +1
> >>>>>>>>
> >>>>>>>> (D) sounds good. What version are you imagining the bundle would
> be,
> >> if
> >>>>>> one runs `br catalog add ~/my/project/ --name
> com.example.myproject`?
> >>>>>>>> ---
> >>>>>>>> I like the idea of uploading a plain zip (rather than only
> >> supporting
> >>>>>> OSGi bundles) - that makes it simpler for non-java folk. The use of
> >> OSGi
> >>>>>> becomes a (hidden) implementation detail to many users.
> >>>>>>>> ---
> >>>>>>>> If auto-generating the manifest, I think we need the user to be
> >>>>> explicit
> >>>>>> about symbolic name and version. Having these supplied in the REST
> api
> >>>>> call
> >>>>>> (as Alex suggests) would achieve that.
> >>>>>>>> Svet, if instead we tried to infer it from the catalog.bom, would
> we
> >>>>>> require some additional metadata within the .bom file? Or would we
> use
> >>>>> the
> >>>>>> catalog item's id + version? I'm not convinced by the latter - it
> >> would
> >>>>>> mean some .bom files would work and others wouldn't (e.g. if the
> .bom
> >> had
> >>>>>> multiple items with different versions). Better to support the
> >> explicit
> >>>>>> approach IMO.
> >>>>>>>> ---
> >>>>>>>> For E ("have a mechanism whereby deployed entities based on an
> >> affected
> >>>>>> blueprint are optionally migrated to the new code"), that feels
> like a
> >>>>>> separate discussion. It could equally apply to a pure YAML .bom file
> >> that
> >>>>>> has been added to the catalog.
> >>>>>>>> I suggest we discuss that in a separate email thread.
> >>>>>>>>
> >>>>>>>> ---
> >>>>>>>> For (G), it's an interesting suggestion from Svet to make use of
> >> Karaf
> >>>>>> Cellar for HA nodes. I'm hesitant (e.g. if restarting a standalone
> >>>>> Brooklyn
> >>>>>> node whose VM has died, then it adds big additional requirements for
> >> what
> >>>>>> constitutes the "persisted state"). On the other hand, it's good to
> >> use
> >>>>>> well-established technologies rather than re-inventing things!
> >>>>>>>> An alternative ("pure brooklyn") approach could be to write the
> >> bundle
> >>>>>> to persisted state; on rebind, we'd install + activate those
> bundles.
> >>>>>>>> ---
> >>>>>>>> For "catalogGroupId", I agree with Svet that in the initial
> use-case
> >>>>>> this can be an implementation detail.
> >>>>>>>> It could be set as the bundle's symbolic name + version:
> everything
> >>>>> from
> >>>>>> the bundle should be deleted at once, along with the bundle.
> >>>>>>>> Longer term, I can see how exposing "catalogGroupId" to the user
> >> could
> >>>>>> support more use-cases (e.g. for several catalog items from
> different
> >>>>>> bundles to work together). I don't think we should try to support
> that
> >>>>> yet.
> >>>>>>>> Aled
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 19/12/2016 17:19, Geoff Macartney wrote:
> >>>>>>>>> hi Alex,
> >>>>>>>>>
> >>>>>>>>> this looks like a good feature to have, I shall look at the PR as
> >> soon
> >>>>>> as I
> >>>>>>>>> can.
> >>>>>>>>>
> >>>>>>>>> The catalog.bom scanner feature was initially enabled by default,
> >> but
> >>>>> we
> >>>>>>>>> had to
> >>>>>>>>> disable it because it turned out not to work properly with
> rebind.
> >> I
> >>>>>> don't
> >>>>>>>>> think
> >>>>>>>>> it should be a lot of work to fix that but it hasn't been
> something
> >>>>>> we've
> >>>>>>>>> got round
> >>>>>>>>> to yet.  This would be a great opportunity to look back at that.
> >>>>>>>>>
> >>>>>>>>> Some random thoughts:
> >>>>>>>>>
> >>>>>>>>> re (C), if we are going to treat the zips as bundles, my gut feel
> >> is
> >>>>>> that
> >>>>>>>>> we
> >>>>>>>>> should insist on a manifest and get the metadata from it.  It
> >> doesn't
> >>>>>> feel
> >>>>>>>>> to me
> >>>>>>>>> like it makes much sense to allow a zip file without a
> MANIFEST.MF
> >> but
> >>>>>>>>> convey
> >>>>>>>>> the intended bundle metadata to Brooklyn via HTTP headers.  And
> >> rather
> >>>>>> than
> >>>>>>>>> infer bundle metadata I think it's better to ask users to be
> >> explicit
> >>>>>> about
> >>>>>>>>> what
> >>>>>>>>> their intentions are.  To make users lives easier, we could
> >>>>>>>>> add a command to br to generate the manifest (locally) with
> correct
> >>>>>> syntax,
> >>>>>>>>> so that the manifest is in the right place, rather than have br
> add
> >>>>> the
> >>>>>> data
> >>>>>>>>> to the "upload" request headers.
> >>>>>>>>>
> >>>>>>>>> re. (D) will be glad to have a look at it
> >>>>>>>>>
> >>>>>>>>> re. (E) it would certainly need to be optional - maybe keep it as
> >> an
> >>>>>>>>> explicit
> >>>>>>>>> separate command ('upgrade'?)
> >>>>>>>>>
> >>>>>>>>> (F) it does seem like a lot of work but might be nice for users
> who
> >>>>> are
> >>>>>> not
> >>>>>>>>> keen on command lines.
> >>>>>>>>>
> >>>>>>>>> G - I:  we'll definitely need to pay close attention to
> persistence
> >>>>> and
> >>>>>>>>> rebind;
> >>>>>>>>> I wonder also about HA operation, are there any additional
> >>>>> implications?
> >>>>>>>>> (J) I think it would be good to treat all the files from a jar,
> >> sorry
> >>>>>>>>> bundle,
> >>>>>>>>> as an atomic group - cleaner that way perhaps than allowing
> >>>>>> delete/update
> >>>>>>>>> of
> >>>>>>>>> individual entries from a bundle on a piecemeal basis.  Rest
> >> support
> >>>>> on
> >>>>>>>>> delete
> >>>>>>>>> catalog could warn about related catalog entries being deleted
> and
> >> ask
> >>>>>> for
> >>>>>>>>> a "--force" param to confirm.
> >>>>>>>>>
> >>>>>>>>> Geoff
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Fri, 16 Dec 2016 at 15:24 Svetoslav Neykov <
> >>>>>>>>> svetoslav.neykov@cloudsoftcorp.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> +1
> >>>>>>>>>>
> >>>>>>>>>> Some thoughts:
> >>>>>>>>>>      * (A) add a utility class BundleMaker
> >>>>>>>>>>        Sounds very similar to
> >>>>>>>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles <
> >>>>>>>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles>
> >>>>>>>>>>        Looking at the code it's much more focused on zip files
> so I
> >>>>> guess
> >>>>>>>>>> there's no much overlap, but worth keeping in mind
> >>>>>>>>>>      * (C) accept bundle symbolic name and version
> >>>>>>>>>>        Why require them at all? Could infer them from the
> >> catalog.bom
> >>>>> in
> >>>>>> some
> >>>>>>>>>> way - maybe require those properties to be in there. If not
> >> present
> >>>>> are
> >>>>>>>>>> they really needed?
> >>>>>>>>>>      * (G) Bundles installed via this mechanism are not
> persisted
> >>>>>> currently &
> >>>>>>>>>> (I) We persist the individual catalog items as YAML, so we end
> up
> >>>>> with
> >>>>>> two
> >>>>>>>>>> records
> >>>>>>>>>>        Suggest marking the catalog items coming from bundles as
> >>>>>>>>>> non-persistable. Then try to share the bundles between HA nodes.
> >>>>> (Karaf
> >>>>>>>>>> Cellar?)
> >>>>>>>>>>      * (J) Introduce a catalogGroupId field on catalog items;
> >>>>>>>>>>        Agree this could be useful and I like the idea of
> deleting
> >> the
> >>>>>> bundle
> >>>>>>>>>> altogether with the catalog items. From user's perspective I
> don't
> >>>>> see
> >>>>>> the
> >>>>>>>>>> need for an extra field (i.e. it's an implementation detail).
> >>>>>>>>>>
> >>>>>>>>>> Svet.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> On 16.12.2016 г., at 12:50, Alex Heneveld <
> >>>>>>>>>> alex.heneveld@cloudsoftcorp.com> wrote:
> >>>>>>>>>>> Hi Brooklyners-
> >>>>>>>>>>>
> >>>>>>>>>>> In the code we currently have two routes for users to install
> new
> >>>>>>>>>>> blueprints:
> >>>>>>>>>>>
> >>>>>>>>>>> (1) upload a catalog YAML file to /v1/catalog
> >>>>>>>>>>>
> >>>>>>>>>>> (2) install a bundle with catalog.bom in the root
> >>>>>>>>>>>
> >>>>>>>>>>> The feature (2) is disabled by default, but I'd like to move
> >> towards
> >>>>>>>>>>> enabling it.  This will make it easier to create nicely
> >> structured
> >>>>> BOM
> >>>>>>>>>>> files because scripts etc can be taken out of the BOM, stored
> as
> >>>>>> files in
> >>>>>>>>>>> the same bundle.  (Because URLs of the form
> >>>>>>>>>>> `classpath://scripts/install.sh`  use the bundle's classpath to
> >>>>>> resolve.)
> >>>>>>>>>>> As a first step in #485 [1] I do a few things:
> >>>>>>>>>>>
> >>>>>>>>>>> (A) add a utility class BundleMaker that lets us create and
> >> modify
> >>>>>>>>>>> bundles/zips, to make it easier to do things we might want to
> >> with
> >>>>>>>>>> bundles,
> >>>>>>>>>>> especially for testing
> >>>>>>>>>>>
> >>>>>>>>>>> (B) add an endpoint to the REST API which allows uploading a
> >> bundle
> >>>>>> ZIP
> >>>>>>>>>>> (C) accept bundle symbolic name and version in that REST API to
> >>>>>>>>>> facilitate
> >>>>>>>>>>> uploading non-bundle ZIPs where the OSGi MANIFEST.MF is
> >>>>> automatically
> >>>>>>>>>>> generated
> >>>>>>>>>>>
> >>>>>>>>>>> With this PR, if you have a directory on your local file system
> >> with
> >>>>>>>>>>> scripts and config files, and a BOM which refers to them, you
> can
> >>>>> just
> >>>>>>>>>> ZIP
> >>>>>>>>>>> that up an upload it, specifying the bundle name so that a YAML
> >>>>>> blueprint
> >>>>>>>>>>> author never needs to touch any java-isms.
> >>>>>>>>>>>
> >>>>>>>>>>> Where I see this going is a development workflow where a user
> can
> >>>>> edit
> >>>>>>>>>>> files locally and upload the ZIP to have that installed, and if
> >> they
> >>>>>> make
> >>>>>>>>>>> changes locally they can POST it again to have catalog items
> >> updated
> >>>>>>>>>>> (because default version is a SNAPSHOT).  We could also:
> >>>>>>>>>>>
> >>>>>>>>>>> (D) have `br catalog add ~/my/project/ --name my.project`
> create
> >> a
> >>>>> ZIP
> >>>>>>>>>> and
> >>>>>>>>>>> POST it, with bundle name metadata, so essentially the user's
> >>>>> process
> >>>>>> is
> >>>>>>>>>>> just to run that whenever they make a change
> >>>>>>>>>>>
> >>>>>>>>>>> (E) have a mechanism whereby deployed entities based on an
> >> affected
> >>>>>>>>>>> blueprint are optionally migrated to the new code, so if you've
> >>>>>> changed
> >>>>>>>>>> an
> >>>>>>>>>>> enricher the changes are picked up, or if say a launch.sh
> script
> >> has
> >>>>>>>>>>> changed, a restart will run the new code
> >>>>>>>>>>>
> >>>>>>>>>>> The above are fairly straightforward programmatically (although
> >> good
> >>>>>> user
> >>>>>>>>>>> interaction with (E) needs some thought).  So I think we can
> >> pretty
> >>>>>>>>>> quickly
> >>>>>>>>>>> get to a much smoother dev workflow.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> That's the highlight of this message.  You can jump to the end,
> >>>>> unless
> >>>>>>>>>>> you're interested in some important but low level details...
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> I'm also tempted by:
> >>>>>>>>>>>
> >>>>>>>>>>> (F) Integration with web-based IDE and/or Brooklyn reading and
> >>>>> writing
> >>>>>>>>>>> straight from GitHub -- but this seems like a lot of work and
> I'm
> >>>>> not
> >>>>>>>>>>> convinced it's much better than (D) workflow-wise
> >>>>>>>>>>>
> >>>>>>>>>>> Before we can change (2) to be the default, or start widely
> using
> >>>>> the
> >>>>>>>>>> POST
> >>>>>>>>>>> a ZIP feature, we need to sort out some issues to do with
> >>>>> persistence
> >>>>>> and
> >>>>>>>>>>> reloading:
> >>>>>>>>>>>
> >>>>>>>>>>> (G) Bundles installed via this mechanism are not persisted
> >>>>> currently,
> >>>>>> so
> >>>>>>>>>> if
> >>>>>>>>>>> you move to a different Brooklyn using the same backing store,
> >>>>> you'll
> >>>>>>>>>> lose
> >>>>>>>>>>> those bundles
> >>>>>>>>>>>
> >>>>>>>>>>> (H) On rebind, bundles aren't always activated when needed,
> >> meaning
> >>>>>> items
> >>>>>>>>>>> can't be loaded
> >>>>>>>>>>>
> >>>>>>>>>>> (I) We persist the individual catalog items as YAML, so we end
> up
> >>>>> with
> >>>>>>>>>> two
> >>>>>>>>>>> records — the YAML from the catalog.bom in the bundle, and the
> >> YAML
> >>>>>>>>>>> persisted for the item.  This isn't a problem per se, but
> >> something
> >>>>> to
> >>>>>>>>>>> think about, and some sometimes surprising behaviour.  In
> >> particular
> >>>>>> if
> >>>>>>>>>> you
> >>>>>>>>>>> delete the persisted YAML, the bundle is still there so the
> item
> >> is
> >>>>> no
> >>>>>>>>>>> longer deleted after a full rebind.
> >>>>>>>>>>>
> >>>>>>>>>>> One idea which might be useful is:
> >>>>>>>>>>>
> >>>>>>>>>>> (J) Introduce a catalogGroupId field on catalog items; this
> will
> >> do
> >>>>>> two
> >>>>>>>>>>> things:  if you try to delete an item with such a record,
> you'll
> >> be
> >>>>>>>>>>> encouraged to delete all such items (maybe disallowed to delete
> >> an
> >>>>>>>>>>> individual one), with the effect of deleting the bundle if it
> >> comes
> >>>>>> from
> >>>>>>>>>> a
> >>>>>>>>>>> bundle; and when resolving types we search first for items with
> >> the
> >>>>>> same
> >>>>>>>>>>> catalogGroupId (so that e.g. if I install MyCluster:1.0 and
> >>>>>> MyNode:1.0 in
> >>>>>>>>>>> the same group, the former can refer simply to "MyNode" but if
> I
> >>>>>> install
> >>>>>>>>>> a
> >>>>>>>>>>> 2.0 version of that group, the 1.0 cluster still loads the 1.0
> >> node
> >>>>> --
> >>>>>>>>>> this
> >>>>>>>>>>> has bitten people i the past)
> >>>>>>>>>>>
> >>>>>>>>>>> There is a related Brooklyn upgrade problem worth mentioning,
> >> which
> >>>>>> the
> >>>>>>>>>>> above might help with, where:
> >>>>>>>>>>>
> >>>>>>>>>>> (K) If I migrate from Brooklyn 10 to 11 when it comes out, I'll
> >> no
> >>>>>> longer
> >>>>>>>>>>> have certain entities that were at v10, since we don't include
> >>>>> those;
> >>>>>> an
> >>>>>>>>>>> upgrade could include rules that certain groupIds need to be
> >>>>> updated,
> >>>>>> or
> >>>>>>>>>> it
> >>>>>>>>>>> can search and attempt to automatically apply the updates
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Quite a lot here and we don't need to solve it but I wanted to:
> >>>>>>>>>>>
> >>>>>>>>>>> * Share the current thinking
> >>>>>>>>>>>
> >>>>>>>>>>> * Get opinions on the general dev workflow suggested by (D)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for feedback -- and if we like it help with (D) would be
> >>>>>>>>>> appreciated!
> >>>>>>>>>>> Best
> >>>>>>>>>>> Alex
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> [1] . https://github.com/apache/brooklyn-server/pull/485
> >> --
> >>
> >> Thomas Bouron • Software Engineer @ Cloudsoft Corporation •
> >> http://www.cloudsoftcorp.com/
> >> Github: https://github.com/tbouron
> >> Twitter: https://twitter.com/eltibouron
> >>
> >> --
> >>
> >> Thomas Bouron • Software Engineer @ Cloudsoft Corporation •
> >> http://www.cloudsoftcorp.com/
> >> Github: https://github.com/tbouron
> >> Twitter: https://twitter.com/eltibouron
> >>
>
> --

Thomas Bouron • Software Engineer @ Cloudsoft Corporation •
http://www.cloudsoftcorp.com/
Github: https://github.com/tbouron
Twitter: https://twitter.com/eltibouron

Re: Uploading ZIPs for a better dev workflow

[Aled Sage <al...@gmail.com>]

Hi Thomas, all,

I think "bundle: ..." is actually a much better name to use in the 
catalog.bom than "symbolicName: ...". The latter makes sense to those 
who know OSGi, but otherwise doesn't suggest anything about "bundling".

---

I think we're very close to merging [1] - further improvements can 
continue incrementally in subsequent PRs, rather than all of this having 
to be done in [1].

---

There's a change compared to the mailing list discussion, I think [2,3]. 
Alex has implemented it so that if "symbolicName" and "id" are used at 
the top-level of the catalog.bom, then they *must match*. This seems 
wrong to me. It's conflating the name spaces of OSGi bundles and catalog 
items. I suggest we revert back to (at least how I interpretted) the 
previous mailing list discussion: if "bundle: " is present, then that is 
used; if "bundle" is absent, then we use "id"; if both are absent, then 
we fail.

Aled

p.s. As an aside, we should deprecate (and soon delete) support for 
"symbolicName" in a catalog.bom - one should use id/version. Nowhere in 
our docs to we say symbolicName. It's just yet another example of us 
supporting multiple ways of doing the same thing, where it's not at all 
clear they mean the same thing, and risking confusing users.

[1] https://github.com/apache/brooklyn-server/pull/485
[2] https://github.com/apache/brooklyn-server/pull/485#discussion_r107144368
[3] 
https://github.com/apache/brooklyn-server/pull/485#issuecomment-288052012


On 23/03/2017 10:23, Geoff Macartney wrote:
> some further thoughts:
>
> re. 1 I agree we should discriminate between the "symbolicName" of the
> bundle and use of the term "symbolicName" in the definition of catalog ids.
> (I'm not sure if just putting it into a "manifest" subsection will keep it
> separate, because of the use of "getFirstAs" at
> https://github.com/apache/brooklyn-server/blob/master/core/src/main/java/org/apache/brooklyn/core/catalog/internal/BasicBrooklynCatalog.java#L542
> ).
>
> re. use of "CatalogPopulator" I think we shall want to factor apart
> CatalogBomScanner/CatalogPopulator into separate classes, -
> - CatalogBomScanner to be the management context service listener
> - CatalogPopulator becomes:
> -- CatalogBundleTracker to do the BundleTracker stuff
> -- CatalogBundleLoader - separate out the handling of the loading of the
> "catalog.bom" from a given bundle
>
> and then 485 can use CatalogBundleLoader.  We'll still need to think about
> the delete side of things, and handling errors, as Thomas says.
>
> re. the Swagger issue, an alternative would be that we could just drop
> "createFromYaml" and "createFromArchive" and let "createPoly" continue to
> do as it does, handling either case; maybe best renamed "createFromUpload".
>
> Geoff
>
> On Thu, 23 Mar 2017 at 09:40 Thomas Bouron <th...@cloudsoftcorp.com>
> wrote:
>
>> Hi all.
>>
>> So I spent time to review and test [1] and here are the few issues I bumped
>> into:
>>
>>     1. the use of `symbolicName` for the bundle clashes Brooklyn: the
>>     `BasicBrooklynCatalog` watches this key to define catalog items IDs [2]
>>     2. because of 1. the bundle that I'm trying to install fails. But upon
>>     failure, the bundle is not uninstalled so any subsequent bundle addition
>>     fail with the first error
>>     3. we now have multiple API definitions for the same endpoint (i.e.
>>     /v1/catalog for yaml and ZIP/JAR) While this is perfectly valid, Swagger
>>     does not support it [3]. It means that the Swagger UI displays only the
>>     latest API definition registered, in this case, that's the one to upload
>>     ZIP/JAR.
>>
>> For 1. I can see 2 solutions: using a special section `manifest` where we
>> put the `symbolicName` and `version`, or changing the field name to
>> `bundle` (I prefer option 2)
>> For 2. we need to add some logic to uninstall the bundle on failure into
>> the `CatalogPopulator`, as per as Geoff comment[4]
>> For 3. even the new Swagger spec (called now OpenAPI) does not support
>> multiple definitions on the same endpoint [5]. I'm not sure if we can leave
>> with that. One solution would be to split the endpoint to create specific
>> one, based on the type of upload.
>>
>> [1] https://github.com/apache/brooklyn-server/pull/485
>> [2]
>>
>> https://github.com/apache/brooklyn-server/blob/master/core/src/main/java/org/apache/brooklyn/core/catalog/internal/BasicBrooklynCatalog.java#L542
>> [3] https://github.com/swagger-api/swagger-core/issues/935
>> [4]
>> https://github.com/apache/brooklyn-server/pull/485#discussion_r107482279
>> [5] https://github.com/OAI/OpenAPI-Specification/issues/182
>>
>> On Mon, 20 Mar 2017 at 15:16 Thomas Bouron <
>> thomas.bouron@cloudsoftcorp.com>
>> wrote:
>>
>> Hi Brooklyners.
>>
>> Uploading ZIPs is really something I'm excited to see coming to life so I
>> would like to help to push this forward.
>>
>> I've re-read this email thread and it sounds like we have a consensus on
>> the "one true way" approach, i.e. always requiring the `symbolicName` and
>> `version` when uploading ZIP/JAR (that includes Svet in his previous
>> email).
>>
>> One thing not explicitly settled is the format of bom for those 2 fields:
>> Alex prefers to put these at the root, Aled in a `manifest` section. I
>> don't have any strong opinion on this: I was leaning toward Aled's
>> suggestion but the argument that says this applies to the whole bundle make
>> sense so I now think putting this at the root is probably better. I can see
>> that Alex already updated his PR[1] to this structure. What the people
>> think about this?
>>
>> I'm going to review [1] but are we happy with this consensus? One thing
>> that will need to be addressed is the persistence/rebind issues but this
>> can be tackled in following PRs.
>>
>> Best.
>>
>> [1] https://github.com/apache/brooklyn-server/pull/485
>>
>> On Tue, 7 Mar 2017 at 16:18 Svetoslav Neykov <
>> svetoslav.neykov@cloudsoftcorp.com> wrote:
>>
>> Having quickly scanned the thread I still don't see a reason why the
>> requirement to have a consistent naming for the bundle itself which is not
>> user visible? Could instead have behaviour similar to the machinNamer here.
>>
>> Slight preference for optional symbolicName when it's a jar file. Though I
>> really like Geoff's idea of "one true way". The workflow to create a jar
>> file is very different from creating a zip file (either from a folder or
>> user supplied) so don't think they conflict.
>>
>> Svet.
>>
>>
>>> On 7.03.2017 \u0433., at 17:47, Alex Heneveld <
>> alex.heneveld@cloudsoftcorp.com>
>> wrote:
>>>
>>> Considering the points made I'm pretty sure it's the case that requiring
>> a symbolic name and version in the BOM presents little if any unnecessary
>> burden, in which case I'm persuaded of the "one true way" philosophy.  I
>> really like it.  :)
>>> Can anyone think of a case where it would be a burden?  My thinking is
>> that:
>>> * If you're uploading a ZIP of YAML and scripts and icons, you have a
>> catalog.bom and that's a useful place to require the symname and version
>>> * If your ZIP/JAR also has a MANIFEST.MF the symname and version will be
>> present in it also; the user is duplicating that info (this is the "little"
>> unnecssary burden) but it's tiny and the benefit of one true way dominates;
>> we do require that they match
>>> * If your ZIP/JAR doesn't have a BOM there's no point in uploading it to
>> the catalog endpoint; either it's a JAR but this isn't meant as an "install
>> arbitrary bundles" endpoint, or a non-JAR in which case what are we
>> supposed to do with it once you upload it?
>>> It keeps the contract simple, and the CLI very simple.   `br push
>> /path/to/blueprint/x/`
>>> (But Geoff as a side point I think the CLI should _not_ attempt
>> validation of the input/version -- the server will do that anyway and no
>> benefit in doing it twice, but more code to maintain.)
>>> I'm further persuaded given it is in a file, we should require a
>> version.  I was concerned about needing to supply the version every time on
>> the CLI but this has gone away.
>>> I see no reason for a separate "manifest" section.  Removing this
>> eliminates the prospect of Aled's pathological/"perverted" example.  I
>> favour:
>>>     brooklyn.catalog:
>>>        symbolicName: com.example.Foo
>>>        version: 1.0
>>>        items:
>>>        - id: foo
>>>          version: 3.0   # optional version override
>>>          itemType: entity
>>>          item:
>>>            type: blah
>>>            ...
>>>
>>> That is, the fields symbolic-name and version are _required_ under the
>> `brooklyn.catalog` when POSTing a ZIP to the catalog endpoint, and are used
>> to define the bundle.  The version (and icon, etc) are then inherited by
>> default by items defined therein.  If the ZIP is a JAR which contains a
>> MANIFEST.MF, the bundle information (symbolic name and version) must match
>> exactly.
>>> Sound right?
>>>
>>> Best
>>> Alex
>>>
>>>
>>> On 03/03/2017 09:59, Geoff Macartney wrote:
>>>> It also means clients ('br' at least) can easily validate the upload
>> even
>>>> before execution by checking that all the manifest details are present.
>>>>
>>>> On Fri, 3 Mar 2017 at 09:57 Geoff Macartney <
>>>> geoff.macartney@cloudsoftcorp.com> wrote:
>>>>
>>>>> Just on the last point of "We could also support the example below",
>> I'd
>>>>> say let's not even be that flexible - my feeling is that the more
>> flexible
>>>>> we are, the more confusing it is, and the harder to get right.  If
>> we're
>>>>> going to add this capability let's just have one way to do it; if you
>> are
>>>>> going to do a ZIP upload, you MUST have a catalog.bom, which MUST
>> contain a
>>>>> manifest section, which MUST have both a symbolic name and a version.
>> That
>>>>> way everything's explicit and very clear, every time.
>>>>>
>>>>>
>>>>>
>>>>> On Thu, 2 Mar 2017 at 11:45 Aled Sage <al...@gmail.com> wrote:
>>>>>
>>>>> Geoff, all,
>>>>>
>>>>> I was imagining the manifest version (in the catalog.bom) to be
>> separate
>>>>> from the item version. The reason is that we support multiple items in
>>>>> the bom that can be independently versioned.
>>>>>
>>>>> Somone perverted could write:
>>>>>
>>>>>      brooklyn.catalog:
>>>>>         version: 1.0
>>>>>         manifest:
>>>>>           symbolic_name: com.example.Foo
>>>>>           version: 2.0
>>>>>         items:
>>>>>         - id: foo
>>>>>           version: 3.0
>>>>>           itemType: entity
>>>>>           item:
>>>>>             type: blah
>>>>>         - id: bar
>>>>>           version: 4.0
>>>>>           itemType: entity
>>>>>           item:
>>>>>          type: blah.blah
>>>>>
>>>>> Here, the "1.0" version is not used by anything; the auto-generated
>> OSGi
>>>>> bundle would be version 2.0; catalog item foo would be 3.0; and catalog
>>>>> item bar would be 4.0.
>>>>>
>>>>> If we were starting from scratch, I'd be tempted to lock things down a
>>>>> lot more for what we actually support. For now, it feels like asking
>> for
>>>>> an explicit version is a reasonable thing to do.
>>>>>
>>>>> ---
>>>>> We could also support the example below, where "1.0" is used by both
>> the
>>>>> manifest and the item:
>>>>>
>>>>>      brooklyn.catalog:
>>>>>         version: 1.0
>>>>>         manifest:
>>>>>           symbolic_name: com.example.Foo
>>>>>         items:
>>>>>         - id: foo
>>>>>           itemType: entity
>>>>>           item:
>>>>>             type: blah
>>>>>
>>>>> The general rule would be: if there is a version in the manifest
>>>>> section, that will be used; if not then a top-level version number will
>>>>> be used. If that is also missing, then fail.
>>>>>
>>>>> i.e. we would not accept:
>>>>>
>>>>>      # Fails - manifest has no version.
>>>>>      brooklyn.catalog:
>>>>>         manifest:
>>>>>           symbolic_name: com.example.Foo
>>>>>         items:
>>>>>         - id: foo
>>>>>           version: 1.0
>>>>>           itemType: entity
>>>>>           item:
>>>>>             type: blah
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> Aled
>>>>>
>>>>>
>>>>> On 02/03/2017 10:02, Geoff Macartney wrote:
>>>>>> I take Alex's point about the manifest being Java specific, and I
>> agree
>>>>>> therefore we shouldn't insist on it.
>>>>>>
>>>>>> +1 also to preferring explicit name/version in the catalog.bom rather
>>>>> than
>>>>>> as API params, I agree we do
>>>>>> need to keep the version in source control.
>>>>>>
>>>>>> Question on your straw man, does the 'version' below
>>>>>>
>>>>>>       brooklyn.catalog:
>>>>>>          manifest:
>>>>>>            symbolic_name: com.example.Foo
>>>>>>            version: 1.0.0-SNAPSHOT
>>>>>>          items:
>>>>>>
>>>>>> _replace_ the 'version' in existing catalogs
>>>>>>
>>>>>> brooklyn.catalog:
>>>>>>       version: "0.1.0-SNAPSHOT"
>>>>>>
>>>>>> or is it an independently variable value, i.e. the version of the
>> bundle,
>>>>>> separate from the version of catalog items that it happens to contain?
>>>>>> (Sounds a bit disquieting, but I guess it's possible).
>>>>>>
>>>>>> e.g. could you have
>>>>>>
>>>>>> brooklyn.catalog:
>>>>>>     version: 0.11.0-SNAPSHOT
>>>>>>     manifest:
>>>>>>       symbolic_name: com.example.Foo
>>>>>>       version: 1.0.0-SNAPSHOT
>>>>>>     items:
>>>>>>     - id: foo
>>>>>>       type: xyz
>>>>>>
>>>>>> which would be bundle com.example.Foo:1.0.0-SNAPSHOT, which happens to
>>>>>> contain item foo:0.11.0-SNAPSHOT?
>>>>>>
>>>>>> If so, what would happen if you left out the second line above? What
>>>>>> version of 'foo' would the catalog contain?
>>>>>>
>>>>>> Or does the version within the manifest _mean_ that it is not only the
>>>>>> bundle version you are specifying, but the catalog item versions too?
>> (I
>>>>>> guess unless the item explicitly supplies its own version.)
>>>>>>
>>>>>> Geoff
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, 1 Mar 2017 at 17:26 Aled Sage <al...@gmail.com> wrote:
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> Discussion of the REST api kicked off again in
>>>>>>
>> https://github.com/apache/brooklyn-server/pull/485#issuecomment-283280366:
>>>>>>       Alex wrote:
>>>>>>
>>>>>>       Requiring a MANIFEST.MF makes the input strongly java-centric;
>> I'd
>>>>>>       like to appeal to people who write YAML blueprints with
>> co-bundled
>>>>>>       scripts and images. They'd wonder why they can't simply make and
>>>>>>       upload a ZIP. They could probably be persuaded to supply a name
>> and
>>>>>>       optional version on a CLI or UI, and understand why it is needed
>>>>>>       (hence supporting those args) but it would not be idiomatic to
>>>>>>       anyone but a java programmer to create a META-INF/ dir with a
>>>>>>       MANIFEST.MF and its syntax. Meanwhile it is very easy for us to
>>>>>>       convert the ZIP to a JAR on the server. Feels like
>> uncontroversial
>>>>>>       good UX.
>>>>>>
>>>>>>       OTOH for a java programmer a MANIFEST.MF is natural, and they'd
>> want
>>>>>>       to drop the name/version args if they are in that file, and I see
>> no
>>>>>>       reason to forbid that pattern.
>>>>>>
>>>>>> I agree with Alex, that we should not require a META-INF/MANIFEST.MF.
>> As
>>>>>> for Geoff's suggestion that we could auto-generate a manifest in the
>>>>>> `br` CLI, I'd prefer a more general solution that works for users of
>> the
>>>>>> REST api as well (i.e. doing it server-side).
>>>>>>
>>>>>> ---
>>>>>> Svet suggested that the catalog.bom could give the symbolic name +
>>>>>> version, via additional metadata in that file.
>>>>>>
>>>>>> What I really like about that is it's in version control, becuase it's
>>>>>> in the file/repo. If alternatively the name/version are just passed as
>>>>>> REST api parameters, then it's not in version control (and is more
>>>>>> susceptible to typos etc).
>>>>>>
>>>>>> I'm not sure what we'd want this to look like. As a straw man:
>>>>>>
>>>>>>       brooklyn.catalog:
>>>>>>          manifest:
>>>>>>            symbolic_name: com.example.Foo
>>>>>>            version: 1.0.0-SNAPSHOT
>>>>>>          items:
>>>>>>          - ...
>>>>>>
>>>>>> If the user built their own real OSGi bundle, then they wouldn't need
>> to
>>>>>> include this "manifest" section. If they did include that and it
>>>>>> contradicted the OSGi bundle's manifest, then we'd fail-fast.
>>>>>>
>>>>>> (Note this reminds me of the (unrelated) metadata described in
>>>>>> https://github.com/brooklyncentral/blueprint-repository- there is a
>>>>>> "publish" block that can be added to the catalog.bom.)
>>>>>>
>>>>>> ---
>>>>>> With Svet's suggestion, if there was no manifest section/file, then
>>>>>> should we auto-generate something from the item(s) in the catalog.bom?
>>>>>>
>>>>>> I can see how that could easily work for a .bom file that has a single
>>>>>> item (e.g. we use the catalog item's id + version, possibly with a
>>>>>> special prefix in the symbolic name to avoid conflicts).
>>>>>>
>>>>>> However, if there are multiple items then it would get trickier.
>>>>>>
>>>>>> I'm inclined to say that for a minimal viable product (MVP) we can
>>>>>> insist on the "manifest" section in the catalog.bom.
>>>>>>
>>>>>> Aled
>>>>>>
>>>>>>
>>>>>> On 20/12/2016 16:34, Svetoslav Neykov wrote:
>>>>>>>> Svet, if instead we tried to infer it from the catalog.bom, would we
>>>>>> require some additional metadata within the .bom file? Or would we use
>>>>> the
>>>>>> catalog item's id + version? I'm not convinced by the latter - it
>> would
>>>>>> mean some .bom files would work and others wouldn't (e.g. if the .bom
>> had
>>>>>> multiple items with different versions). Better to support the
>> explicit
>>>>>> approach IMO.
>>>>>>> I imagine it would be additional metadata. On the other hand I don't
>> see
>>>>>> a technical reason why we need an explicit symbolicName and version -
>>>>> they
>>>>>> can be auto-generated.
>>>>>>> Svet.
>>>>>>>
>>>>>>>
>>>>>>>> On 20.12.2016 \u0433., at 17:50, Aled Sage <al...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> +1
>>>>>>>>
>>>>>>>> (D) sounds good. What version are you imagining the bundle would be,
>> if
>>>>>> one runs `br catalog add ~/my/project/ --name com.example.myproject`?
>>>>>>>> ---
>>>>>>>> I like the idea of uploading a plain zip (rather than only
>> supporting
>>>>>> OSGi bundles) - that makes it simpler for non-java folk. The use of
>> OSGi
>>>>>> becomes a (hidden) implementation detail to many users.
>>>>>>>> ---
>>>>>>>> If auto-generating the manifest, I think we need the user to be
>>>>> explicit
>>>>>> about symbolic name and version. Having these supplied in the REST api
>>>>> call
>>>>>> (as Alex suggests) would achieve that.
>>>>>>>> Svet, if instead we tried to infer it from the catalog.bom, would we
>>>>>> require some additional metadata within the .bom file? Or would we use
>>>>> the
>>>>>> catalog item's id + version? I'm not convinced by the latter - it
>> would
>>>>>> mean some .bom files would work and others wouldn't (e.g. if the .bom
>> had
>>>>>> multiple items with different versions). Better to support the
>> explicit
>>>>>> approach IMO.
>>>>>>>> ---
>>>>>>>> For E ("have a mechanism whereby deployed entities based on an
>> affected
>>>>>> blueprint are optionally migrated to the new code"), that feels like a
>>>>>> separate discussion. It could equally apply to a pure YAML .bom file
>> that
>>>>>> has been added to the catalog.
>>>>>>>> I suggest we discuss that in a separate email thread.
>>>>>>>>
>>>>>>>> ---
>>>>>>>> For (G), it's an interesting suggestion from Svet to make use of
>> Karaf
>>>>>> Cellar for HA nodes. I'm hesitant (e.g. if restarting a standalone
>>>>> Brooklyn
>>>>>> node whose VM has died, then it adds big additional requirements for
>> what
>>>>>> constitutes the "persisted state"). On the other hand, it's good to
>> use
>>>>>> well-established technologies rather than re-inventing things!
>>>>>>>> An alternative ("pure brooklyn") approach could be to write the
>> bundle
>>>>>> to persisted state; on rebind, we'd install + activate those bundles.
>>>>>>>> ---
>>>>>>>> For "catalogGroupId", I agree with Svet that in the initial use-case
>>>>>> this can be an implementation detail.
>>>>>>>> It could be set as the bundle's symbolic name + version: everything
>>>>> from
>>>>>> the bundle should be deleted at once, along with the bundle.
>>>>>>>> Longer term, I can see how exposing "catalogGroupId" to the user
>> could
>>>>>> support more use-cases (e.g. for several catalog items from different
>>>>>> bundles to work together). I don't think we should try to support that
>>>>> yet.
>>>>>>>> Aled
>>>>>>>>
>>>>>>>>
>>>>>>>> On 19/12/2016 17:19, Geoff Macartney wrote:
>>>>>>>>> hi Alex,
>>>>>>>>>
>>>>>>>>> this looks like a good feature to have, I shall look at the PR as
>> soon
>>>>>> as I
>>>>>>>>> can.
>>>>>>>>>
>>>>>>>>> The catalog.bom scanner feature was initially enabled by default,
>> but
>>>>> we
>>>>>>>>> had to
>>>>>>>>> disable it because it turned out not to work properly with rebind.
>> I
>>>>>> don't
>>>>>>>>> think
>>>>>>>>> it should be a lot of work to fix that but it hasn't been something
>>>>>> we've
>>>>>>>>> got round
>>>>>>>>> to yet.  This would be a great opportunity to look back at that.
>>>>>>>>>
>>>>>>>>> Some random thoughts:
>>>>>>>>>
>>>>>>>>> re (C), if we are going to treat the zips as bundles, my gut feel
>> is
>>>>>> that
>>>>>>>>> we
>>>>>>>>> should insist on a manifest and get the metadata from it.  It
>> doesn't
>>>>>> feel
>>>>>>>>> to me
>>>>>>>>> like it makes much sense to allow a zip file without a MANIFEST.MF
>> but
>>>>>>>>> convey
>>>>>>>>> the intended bundle metadata to Brooklyn via HTTP headers.  And
>> rather
>>>>>> than
>>>>>>>>> infer bundle metadata I think it's better to ask users to be
>> explicit
>>>>>> about
>>>>>>>>> what
>>>>>>>>> their intentions are.  To make users lives easier, we could
>>>>>>>>> add a command to br to generate the manifest (locally) with correct
>>>>>> syntax,
>>>>>>>>> so that the manifest is in the right place, rather than have br add
>>>>> the
>>>>>> data
>>>>>>>>> to the "upload" request headers.
>>>>>>>>>
>>>>>>>>> re. (D) will be glad to have a look at it
>>>>>>>>>
>>>>>>>>> re. (E) it would certainly need to be optional - maybe keep it as
>> an
>>>>>>>>> explicit
>>>>>>>>> separate command ('upgrade'?)
>>>>>>>>>
>>>>>>>>> (F) it does seem like a lot of work but might be nice for users who
>>>>> are
>>>>>> not
>>>>>>>>> keen on command lines.
>>>>>>>>>
>>>>>>>>> G - I:  we'll definitely need to pay close attention to persistence
>>>>> and
>>>>>>>>> rebind;
>>>>>>>>> I wonder also about HA operation, are there any additional
>>>>> implications?
>>>>>>>>> (J) I think it would be good to treat all the files from a jar,
>> sorry
>>>>>>>>> bundle,
>>>>>>>>> as an atomic group - cleaner that way perhaps than allowing
>>>>>> delete/update
>>>>>>>>> of
>>>>>>>>> individual entries from a bundle on a piecemeal basis.  Rest
>> support
>>>>> on
>>>>>>>>> delete
>>>>>>>>> catalog could warn about related catalog entries being deleted and
>> ask
>>>>>> for
>>>>>>>>> a "--force" param to confirm.
>>>>>>>>>
>>>>>>>>> Geoff
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, 16 Dec 2016 at 15:24 Svetoslav Neykov <
>>>>>>>>> svetoslav.neykov@cloudsoftcorp.com> wrote:
>>>>>>>>>
>>>>>>>>>> +1
>>>>>>>>>>
>>>>>>>>>> Some thoughts:
>>>>>>>>>>      * (A) add a utility class BundleMaker
>>>>>>>>>>        Sounds very similar to
>>>>>>>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles <
>>>>>>>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles>
>>>>>>>>>>        Looking at the code it's much more focused on zip files so I
>>>>> guess
>>>>>>>>>> there's no much overlap, but worth keeping in mind
>>>>>>>>>>      * (C) accept bundle symbolic name and version
>>>>>>>>>>        Why require them at all? Could infer them from the
>> catalog.bom
>>>>> in
>>>>>> some
>>>>>>>>>> way - maybe require those properties to be in there. If not
>> present
>>>>> are
>>>>>>>>>> they really needed?
>>>>>>>>>>      * (G) Bundles installed via this mechanism are not persisted
>>>>>> currently &
>>>>>>>>>> (I) We persist the individual catalog items as YAML, so we end up
>>>>> with
>>>>>> two
>>>>>>>>>> records
>>>>>>>>>>        Suggest marking the catalog items coming from bundles as
>>>>>>>>>> non-persistable. Then try to share the bundles between HA nodes.
>>>>> (Karaf
>>>>>>>>>> Cellar?)
>>>>>>>>>>      * (J) Introduce a catalogGroupId field on catalog items;
>>>>>>>>>>        Agree this could be useful and I like the idea of deleting
>> the
>>>>>> bundle
>>>>>>>>>> altogether with the catalog items. From user's perspective I don't
>>>>> see
>>>>>> the
>>>>>>>>>> need for an extra field (i.e. it's an implementation detail).
>>>>>>>>>>
>>>>>>>>>> Svet.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On 16.12.2016 \u0433., at 12:50, Alex Heneveld <
>>>>>>>>>> alex.heneveld@cloudsoftcorp.com> wrote:
>>>>>>>>>>> Hi Brooklyners-
>>>>>>>>>>>
>>>>>>>>>>> In the code we currently have two routes for users to install new
>>>>>>>>>>> blueprints:
>>>>>>>>>>>
>>>>>>>>>>> (1) upload a catalog YAML file to /v1/catalog
>>>>>>>>>>>
>>>>>>>>>>> (2) install a bundle with catalog.bom in the root
>>>>>>>>>>>
>>>>>>>>>>> The feature (2) is disabled by default, but I'd like to move
>> towards
>>>>>>>>>>> enabling it.  This will make it easier to create nicely
>> structured
>>>>> BOM
>>>>>>>>>>> files because scripts etc can be taken out of the BOM, stored as
>>>>>> files in
>>>>>>>>>>> the same bundle.  (Because URLs of the form
>>>>>>>>>>> `classpath://scripts/install.sh`  use the bundle's classpath to
>>>>>> resolve.)
>>>>>>>>>>> As a first step in #485 [1] I do a few things:
>>>>>>>>>>>
>>>>>>>>>>> (A) add a utility class BundleMaker that lets us create and
>> modify
>>>>>>>>>>> bundles/zips, to make it easier to do things we might want to
>> with
>>>>>>>>>> bundles,
>>>>>>>>>>> especially for testing
>>>>>>>>>>>
>>>>>>>>>>> (B) add an endpoint to the REST API which allows uploading a
>> bundle
>>>>>> ZIP
>>>>>>>>>>> (C) accept bundle symbolic name and version in that REST API to
>>>>>>>>>> facilitate
>>>>>>>>>>> uploading non-bundle ZIPs where the OSGi MANIFEST.MF is
>>>>> automatically
>>>>>>>>>>> generated
>>>>>>>>>>>
>>>>>>>>>>> With this PR, if you have a directory on your local file system
>> with
>>>>>>>>>>> scripts and config files, and a BOM which refers to them, you can
>>>>> just
>>>>>>>>>> ZIP
>>>>>>>>>>> that up an upload it, specifying the bundle name so that a YAML
>>>>>> blueprint
>>>>>>>>>>> author never needs to touch any java-isms.
>>>>>>>>>>>
>>>>>>>>>>> Where I see this going is a development workflow where a user can
>>>>> edit
>>>>>>>>>>> files locally and upload the ZIP to have that installed, and if
>> they
>>>>>> make
>>>>>>>>>>> changes locally they can POST it again to have catalog items
>> updated
>>>>>>>>>>> (because default version is a SNAPSHOT).  We could also:
>>>>>>>>>>>
>>>>>>>>>>> (D) have `br catalog add ~/my/project/ --name my.project` create
>> a
>>>>> ZIP
>>>>>>>>>> and
>>>>>>>>>>> POST it, with bundle name metadata, so essentially the user's
>>>>> process
>>>>>> is
>>>>>>>>>>> just to run that whenever they make a change
>>>>>>>>>>>
>>>>>>>>>>> (E) have a mechanism whereby deployed entities based on an
>> affected
>>>>>>>>>>> blueprint are optionally migrated to the new code, so if you've
>>>>>> changed
>>>>>>>>>> an
>>>>>>>>>>> enricher the changes are picked up, or if say a launch.sh script
>> has
>>>>>>>>>>> changed, a restart will run the new code
>>>>>>>>>>>
>>>>>>>>>>> The above are fairly straightforward programmatically (although
>> good
>>>>>> user
>>>>>>>>>>> interaction with (E) needs some thought).  So I think we can
>> pretty
>>>>>>>>>> quickly
>>>>>>>>>>> get to a much smoother dev workflow.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> That's the highlight of this message.  You can jump to the end,
>>>>> unless
>>>>>>>>>>> you're interested in some important but low level details...
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I'm also tempted by:
>>>>>>>>>>>
>>>>>>>>>>> (F) Integration with web-based IDE and/or Brooklyn reading and
>>>>> writing
>>>>>>>>>>> straight from GitHub -- but this seems like a lot of work and I'm
>>>>> not
>>>>>>>>>>> convinced it's much better than (D) workflow-wise
>>>>>>>>>>>
>>>>>>>>>>> Before we can change (2) to be the default, or start widely using
>>>>> the
>>>>>>>>>> POST
>>>>>>>>>>> a ZIP feature, we need to sort out some issues to do with
>>>>> persistence
>>>>>> and
>>>>>>>>>>> reloading:
>>>>>>>>>>>
>>>>>>>>>>> (G) Bundles installed via this mechanism are not persisted
>>>>> currently,
>>>>>> so
>>>>>>>>>> if
>>>>>>>>>>> you move to a different Brooklyn using the same backing store,
>>>>> you'll
>>>>>>>>>> lose
>>>>>>>>>>> those bundles
>>>>>>>>>>>
>>>>>>>>>>> (H) On rebind, bundles aren't always activated when needed,
>> meaning
>>>>>> items
>>>>>>>>>>> can't be loaded
>>>>>>>>>>>
>>>>>>>>>>> (I) We persist the individual catalog items as YAML, so we end up
>>>>> with
>>>>>>>>>> two
>>>>>>>>>>> records \u2014 the YAML from the catalog.bom in the bundle, and the
>> YAML
>>>>>>>>>>> persisted for the item.  This isn't a problem per se, but
>> something
>>>>> to
>>>>>>>>>>> think about, and some sometimes surprising behaviour.  In
>> particular
>>>>>> if
>>>>>>>>>> you
>>>>>>>>>>> delete the persisted YAML, the bundle is still there so the item
>> is
>>>>> no
>>>>>>>>>>> longer deleted after a full rebind.
>>>>>>>>>>>
>>>>>>>>>>> One idea which might be useful is:
>>>>>>>>>>>
>>>>>>>>>>> (J) Introduce a catalogGroupId field on catalog items; this will
>> do
>>>>>> two
>>>>>>>>>>> things:  if you try to delete an item with such a record, you'll
>> be
>>>>>>>>>>> encouraged to delete all such items (maybe disallowed to delete
>> an
>>>>>>>>>>> individual one), with the effect of deleting the bundle if it
>> comes
>>>>>> from
>>>>>>>>>> a
>>>>>>>>>>> bundle; and when resolving types we search first for items with
>> the
>>>>>> same
>>>>>>>>>>> catalogGroupId (so that e.g. if I install MyCluster:1.0 and
>>>>>> MyNode:1.0 in
>>>>>>>>>>> the same group, the former can refer simply to "MyNode" but if I
>>>>>> install
>>>>>>>>>> a
>>>>>>>>>>> 2.0 version of that group, the 1.0 cluster still loads the 1.0
>> node
>>>>> --
>>>>>>>>>> this
>>>>>>>>>>> has bitten people i the past)
>>>>>>>>>>>
>>>>>>>>>>> There is a related Brooklyn upgrade problem worth mentioning,
>> which
>>>>>> the
>>>>>>>>>>> above might help with, where:
>>>>>>>>>>>
>>>>>>>>>>> (K) If I migrate from Brooklyn 10 to 11 when it comes out, I'll
>> no
>>>>>> longer
>>>>>>>>>>> have certain entities that were at v10, since we don't include
>>>>> those;
>>>>>> an
>>>>>>>>>>> upgrade could include rules that certain groupIds need to be
>>>>> updated,
>>>>>> or
>>>>>>>>>> it
>>>>>>>>>>> can search and attempt to automatically apply the updates
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Quite a lot here and we don't need to solve it but I wanted to:
>>>>>>>>>>>
>>>>>>>>>>> * Share the current thinking
>>>>>>>>>>>
>>>>>>>>>>> * Get opinions on the general dev workflow suggested by (D)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks for feedback -- and if we like it help with (D) would be
>>>>>>>>>> appreciated!
>>>>>>>>>>> Best
>>>>>>>>>>> Alex
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> [1] . https://github.com/apache/brooklyn-server/pull/485
>> --
>>
>> Thomas Bouron \u2022 Software Engineer @ Cloudsoft Corporation \u2022
>> http://www.cloudsoftcorp.com/
>> Github: https://github.com/tbouron
>> Twitter: https://twitter.com/eltibouron
>>
>> --
>>
>> Thomas Bouron \u2022 Software Engineer @ Cloudsoft Corporation \u2022
>> http://www.cloudsoftcorp.com/
>> Github: https://github.com/tbouron
>> Twitter: https://twitter.com/eltibouron
>>

Re: Uploading ZIPs for a better dev workflow

[Geoff Macartney <ge...@cloudsoftcorp.com>]

some further thoughts:

re. 1 I agree we should discriminate between the "symbolicName" of the
bundle and use of the term "symbolicName" in the definition of catalog ids.
(I'm not sure if just putting it into a "manifest" subsection will keep it
separate, because of the use of "getFirstAs" at
https://github.com/apache/brooklyn-server/blob/master/core/src/main/java/org/apache/brooklyn/core/catalog/internal/BasicBrooklynCatalog.java#L542
).

re. use of "CatalogPopulator" I think we shall want to factor apart
CatalogBomScanner/CatalogPopulator into separate classes, -
- CatalogBomScanner to be the management context service listener
- CatalogPopulator becomes:
-- CatalogBundleTracker to do the BundleTracker stuff
-- CatalogBundleLoader - separate out the handling of the loading of the
"catalog.bom" from a given bundle

and then 485 can use CatalogBundleLoader.  We'll still need to think about
the delete side of things, and handling errors, as Thomas says.

re. the Swagger issue, an alternative would be that we could just drop
"createFromYaml" and "createFromArchive" and let "createPoly" continue to
do as it does, handling either case; maybe best renamed "createFromUpload".

Geoff

On Thu, 23 Mar 2017 at 09:40 Thomas Bouron <th...@cloudsoftcorp.com>
wrote:

> Hi all.
>
> So I spent time to review and test [1] and here are the few issues I bumped
> into:
>
>    1. the use of `symbolicName` for the bundle clashes Brooklyn: the
>    `BasicBrooklynCatalog` watches this key to define catalog items IDs [2]
>    2. because of 1. the bundle that I'm trying to install fails. But upon
>    failure, the bundle is not uninstalled so any subsequent bundle addition
>    fail with the first error
>    3. we now have multiple API definitions for the same endpoint (i.e.
>    /v1/catalog for yaml and ZIP/JAR) While this is perfectly valid, Swagger
>    does not support it [3]. It means that the Swagger UI displays only the
>    latest API definition registered, in this case, that's the one to upload
>    ZIP/JAR.
>
> For 1. I can see 2 solutions: using a special section `manifest` where we
> put the `symbolicName` and `version`, or changing the field name to
> `bundle` (I prefer option 2)
> For 2. we need to add some logic to uninstall the bundle on failure into
> the `CatalogPopulator`, as per as Geoff comment[4]
> For 3. even the new Swagger spec (called now OpenAPI) does not support
> multiple definitions on the same endpoint [5]. I'm not sure if we can leave
> with that. One solution would be to split the endpoint to create specific
> one, based on the type of upload.
>
> [1] https://github.com/apache/brooklyn-server/pull/485
> [2]
>
> https://github.com/apache/brooklyn-server/blob/master/core/src/main/java/org/apache/brooklyn/core/catalog/internal/BasicBrooklynCatalog.java#L542
> [3] https://github.com/swagger-api/swagger-core/issues/935
> [4]
> https://github.com/apache/brooklyn-server/pull/485#discussion_r107482279
> [5] https://github.com/OAI/OpenAPI-Specification/issues/182
>
> On Mon, 20 Mar 2017 at 15:16 Thomas Bouron <
> thomas.bouron@cloudsoftcorp.com>
> wrote:
>
> Hi Brooklyners.
>
> Uploading ZIPs is really something I'm excited to see coming to life so I
> would like to help to push this forward.
>
> I've re-read this email thread and it sounds like we have a consensus on
> the "one true way" approach, i.e. always requiring the `symbolicName` and
> `version` when uploading ZIP/JAR (that includes Svet in his previous
> email).
>
> One thing not explicitly settled is the format of bom for those 2 fields:
> Alex prefers to put these at the root, Aled in a `manifest` section. I
> don't have any strong opinion on this: I was leaning toward Aled's
> suggestion but the argument that says this applies to the whole bundle make
> sense so I now think putting this at the root is probably better. I can see
> that Alex already updated his PR[1] to this structure. What the people
> think about this?
>
> I'm going to review [1] but are we happy with this consensus? One thing
> that will need to be addressed is the persistence/rebind issues but this
> can be tackled in following PRs.
>
> Best.
>
> [1] https://github.com/apache/brooklyn-server/pull/485
>
> On Tue, 7 Mar 2017 at 16:18 Svetoslav Neykov <
> svetoslav.neykov@cloudsoftcorp.com> wrote:
>
> Having quickly scanned the thread I still don't see a reason why the
> requirement to have a consistent naming for the bundle itself which is not
> user visible? Could instead have behaviour similar to the machinNamer here.
>
> Slight preference for optional symbolicName when it's a jar file. Though I
> really like Geoff's idea of "one true way". The workflow to create a jar
> file is very different from creating a zip file (either from a folder or
> user supplied) so don't think they conflict.
>
> Svet.
>
>
> > On 7.03.2017 г., at 17:47, Alex Heneveld <
> alex.heneveld@cloudsoftcorp.com>
> wrote:
> >
> >
> > Considering the points made I'm pretty sure it's the case that requiring
> a symbolic name and version in the BOM presents little if any unnecessary
> burden, in which case I'm persuaded of the "one true way" philosophy.  I
> really like it.  :)
> >
> > Can anyone think of a case where it would be a burden?  My thinking is
> that:
> >
> > * If you're uploading a ZIP of YAML and scripts and icons, you have a
> catalog.bom and that's a useful place to require the symname and version
> > * If your ZIP/JAR also has a MANIFEST.MF the symname and version will be
> present in it also; the user is duplicating that info (this is the "little"
> unnecssary burden) but it's tiny and the benefit of one true way dominates;
> we do require that they match
> > * If your ZIP/JAR doesn't have a BOM there's no point in uploading it to
> the catalog endpoint; either it's a JAR but this isn't meant as an "install
> arbitrary bundles" endpoint, or a non-JAR in which case what are we
> supposed to do with it once you upload it?
> >
> > It keeps the contract simple, and the CLI very simple.   `br push
> /path/to/blueprint/x/`
> > (But Geoff as a side point I think the CLI should _not_ attempt
> validation of the input/version -- the server will do that anyway and no
> benefit in doing it twice, but more code to maintain.)
> >
> > I'm further persuaded given it is in a file, we should require a
> version.  I was concerned about needing to supply the version every time on
> the CLI but this has gone away.
> >
> > I see no reason for a separate "manifest" section.  Removing this
> eliminates the prospect of Aled's pathological/"perverted" example.  I
> favour:
> >
> >    brooklyn.catalog:
> >       symbolicName: com.example.Foo
> >       version: 1.0
> >       items:
> >       - id: foo
> >         version: 3.0   # optional version override
> >         itemType: entity
> >         item:
> >           type: blah
> >           ...
> >
> > That is, the fields symbolic-name and version are _required_ under the
> `brooklyn.catalog` when POSTing a ZIP to the catalog endpoint, and are used
> to define the bundle.  The version (and icon, etc) are then inherited by
> default by items defined therein.  If the ZIP is a JAR which contains a
> MANIFEST.MF, the bundle information (symbolic name and version) must match
> exactly.
> >
> > Sound right?
> >
> > Best
> > Alex
> >
> >
> > On 03/03/2017 09:59, Geoff Macartney wrote:
> >> It also means clients ('br' at least) can easily validate the upload
> even
> >> before execution by checking that all the manifest details are present.
> >>
> >> On Fri, 3 Mar 2017 at 09:57 Geoff Macartney <
> >> geoff.macartney@cloudsoftcorp.com> wrote:
> >>
> >>> Just on the last point of "We could also support the example below",
> I'd
> >>> say let's not even be that flexible - my feeling is that the more
> flexible
> >>> we are, the more confusing it is, and the harder to get right.  If
> we're
> >>> going to add this capability let's just have one way to do it; if you
> are
> >>> going to do a ZIP upload, you MUST have a catalog.bom, which MUST
> contain a
> >>> manifest section, which MUST have both a symbolic name and a version.
> That
> >>> way everything's explicit and very clear, every time.
> >>>
> >>>
> >>>
> >>> On Thu, 2 Mar 2017 at 11:45 Aled Sage <al...@gmail.com> wrote:
> >>>
> >>> Geoff, all,
> >>>
> >>> I was imagining the manifest version (in the catalog.bom) to be
> separate
> >>> from the item version. The reason is that we support multiple items in
> >>> the bom that can be independently versioned.
> >>>
> >>> Somone perverted could write:
> >>>
> >>>     brooklyn.catalog:
> >>>        version: 1.0
> >>>        manifest:
> >>>          symbolic_name: com.example.Foo
> >>>          version: 2.0
> >>>        items:
> >>>        - id: foo
> >>>          version: 3.0
> >>>          itemType: entity
> >>>          item:
> >>>            type: blah
> >>>        - id: bar
> >>>          version: 4.0
> >>>          itemType: entity
> >>>          item:
> >>>         type: blah.blah
> >>>
> >>> Here, the "1.0" version is not used by anything; the auto-generated
> OSGi
> >>> bundle would be version 2.0; catalog item foo would be 3.0; and catalog
> >>> item bar would be 4.0.
> >>>
> >>> If we were starting from scratch, I'd be tempted to lock things down a
> >>> lot more for what we actually support. For now, it feels like asking
> for
> >>> an explicit version is a reasonable thing to do.
> >>>
> >>> ---
> >>> We could also support the example below, where "1.0" is used by both
> the
> >>> manifest and the item:
> >>>
> >>>     brooklyn.catalog:
> >>>        version: 1.0
> >>>        manifest:
> >>>          symbolic_name: com.example.Foo
> >>>        items:
> >>>        - id: foo
> >>>          itemType: entity
> >>>          item:
> >>>            type: blah
> >>>
> >>> The general rule would be: if there is a version in the manifest
> >>> section, that will be used; if not then a top-level version number will
> >>> be used. If that is also missing, then fail.
> >>>
> >>> i.e. we would not accept:
> >>>
> >>>     # Fails - manifest has no version.
> >>>     brooklyn.catalog:
> >>>        manifest:
> >>>          symbolic_name: com.example.Foo
> >>>        items:
> >>>        - id: foo
> >>>          version: 1.0
> >>>          itemType: entity
> >>>          item:
> >>>            type: blah
> >>>
> >>> Thoughts?
> >>>
> >>> Aled
> >>>
> >>>
> >>> On 02/03/2017 10:02, Geoff Macartney wrote:
> >>>> I take Alex's point about the manifest being Java specific, and I
> agree
> >>>> therefore we shouldn't insist on it.
> >>>>
> >>>> +1 also to preferring explicit name/version in the catalog.bom rather
> >>> than
> >>>> as API params, I agree we do
> >>>> need to keep the version in source control.
> >>>>
> >>>> Question on your straw man, does the 'version' below
> >>>>
> >>>>      brooklyn.catalog:
> >>>>         manifest:
> >>>>           symbolic_name: com.example.Foo
> >>>>           version: 1.0.0-SNAPSHOT
> >>>>         items:
> >>>>
> >>>> _replace_ the 'version' in existing catalogs
> >>>>
> >>>> brooklyn.catalog:
> >>>>      version: "0.1.0-SNAPSHOT"
> >>>>
> >>>> or is it an independently variable value, i.e. the version of the
> bundle,
> >>>> separate from the version of catalog items that it happens to contain?
> >>>> (Sounds a bit disquieting, but I guess it's possible).
> >>>>
> >>>> e.g. could you have
> >>>>
> >>>> brooklyn.catalog:
> >>>>    version: 0.11.0-SNAPSHOT
> >>>>    manifest:
> >>>>      symbolic_name: com.example.Foo
> >>>>      version: 1.0.0-SNAPSHOT
> >>>>    items:
> >>>>    - id: foo
> >>>>      type: xyz
> >>>>
> >>>> which would be bundle com.example.Foo:1.0.0-SNAPSHOT, which happens to
> >>>> contain item foo:0.11.0-SNAPSHOT?
> >>>>
> >>>> If so, what would happen if you left out the second line above? What
> >>>> version of 'foo' would the catalog contain?
> >>>>
> >>>> Or does the version within the manifest _mean_ that it is not only the
> >>>> bundle version you are specifying, but the catalog item versions too?
> (I
> >>>> guess unless the item explicitly supplies its own version.)
> >>>>
> >>>> Geoff
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Wed, 1 Mar 2017 at 17:26 Aled Sage <al...@gmail.com> wrote:
> >>>>
> >>>> Hi all,
> >>>>
> >>>> Discussion of the REST api kicked off again in
> >>>>
> >>>
> https://github.com/apache/brooklyn-server/pull/485#issuecomment-283280366:
> >>>>      Alex wrote:
> >>>>
> >>>>      Requiring a MANIFEST.MF makes the input strongly java-centric;
> I'd
> >>>>      like to appeal to people who write YAML blueprints with
> co-bundled
> >>>>      scripts and images. They'd wonder why they can't simply make and
> >>>>      upload a ZIP. They could probably be persuaded to supply a name
> and
> >>>>      optional version on a CLI or UI, and understand why it is needed
> >>>>      (hence supporting those args) but it would not be idiomatic to
> >>>>      anyone but a java programmer to create a META-INF/ dir with a
> >>>>      MANIFEST.MF and its syntax. Meanwhile it is very easy for us to
> >>>>      convert the ZIP to a JAR on the server. Feels like
> uncontroversial
> >>>>      good UX.
> >>>>
> >>>>      OTOH for a java programmer a MANIFEST.MF is natural, and they'd
> want
> >>>>      to drop the name/version args if they are in that file, and I see
> no
> >>>>      reason to forbid that pattern.
> >>>>
> >>>> I agree with Alex, that we should not require a META-INF/MANIFEST.MF.
> As
> >>>> for Geoff's suggestion that we could auto-generate a manifest in the
> >>>> `br` CLI, I'd prefer a more general solution that works for users of
> the
> >>>> REST api as well (i.e. doing it server-side).
> >>>>
> >>>> ---
> >>>> Svet suggested that the catalog.bom could give the symbolic name +
> >>>> version, via additional metadata in that file.
> >>>>
> >>>> What I really like about that is it's in version control, becuase it's
> >>>> in the file/repo. If alternatively the name/version are just passed as
> >>>> REST api parameters, then it's not in version control (and is more
> >>>> susceptible to typos etc).
> >>>>
> >>>> I'm not sure what we'd want this to look like. As a straw man:
> >>>>
> >>>>      brooklyn.catalog:
> >>>>         manifest:
> >>>>           symbolic_name: com.example.Foo
> >>>>           version: 1.0.0-SNAPSHOT
> >>>>         items:
> >>>>         - ...
> >>>>
> >>>> If the user built their own real OSGi bundle, then they wouldn't need
> to
> >>>> include this "manifest" section. If they did include that and it
> >>>> contradicted the OSGi bundle's manifest, then we'd fail-fast.
> >>>>
> >>>> (Note this reminds me of the (unrelated) metadata described in
> >>>> https://github.com/brooklyncentral/blueprint-repository- there is a
> >>>> "publish" block that can be added to the catalog.bom.)
> >>>>
> >>>> ---
> >>>> With Svet's suggestion, if there was no manifest section/file, then
> >>>> should we auto-generate something from the item(s) in the catalog.bom?
> >>>>
> >>>> I can see how that could easily work for a .bom file that has a single
> >>>> item (e.g. we use the catalog item's id + version, possibly with a
> >>>> special prefix in the symbolic name to avoid conflicts).
> >>>>
> >>>> However, if there are multiple items then it would get trickier.
> >>>>
> >>>> I'm inclined to say that for a minimal viable product (MVP) we can
> >>>> insist on the "manifest" section in the catalog.bom.
> >>>>
> >>>> Aled
> >>>>
> >>>>
> >>>> On 20/12/2016 16:34, Svetoslav Neykov wrote:
> >>>>>> Svet, if instead we tried to infer it from the catalog.bom, would we
> >>>> require some additional metadata within the .bom file? Or would we use
> >>> the
> >>>> catalog item's id + version? I'm not convinced by the latter - it
> would
> >>>> mean some .bom files would work and others wouldn't (e.g. if the .bom
> had
> >>>> multiple items with different versions). Better to support the
> explicit
> >>>> approach IMO.
> >>>>> I imagine it would be additional metadata. On the other hand I don't
> see
> >>>> a technical reason why we need an explicit symbolicName and version -
> >>> they
> >>>> can be auto-generated.
> >>>>> Svet.
> >>>>>
> >>>>>
> >>>>>> On 20.12.2016 г., at 17:50, Aled Sage <al...@gmail.com> wrote:
> >>>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> +1
> >>>>>>
> >>>>>> (D) sounds good. What version are you imagining the bundle would be,
> if
> >>>> one runs `br catalog add ~/my/project/ --name com.example.myproject`?
> >>>>>> ---
> >>>>>> I like the idea of uploading a plain zip (rather than only
> supporting
> >>>> OSGi bundles) - that makes it simpler for non-java folk. The use of
> OSGi
> >>>> becomes a (hidden) implementation detail to many users.
> >>>>>> ---
> >>>>>> If auto-generating the manifest, I think we need the user to be
> >>> explicit
> >>>> about symbolic name and version. Having these supplied in the REST api
> >>> call
> >>>> (as Alex suggests) would achieve that.
> >>>>>> Svet, if instead we tried to infer it from the catalog.bom, would we
> >>>> require some additional metadata within the .bom file? Or would we use
> >>> the
> >>>> catalog item's id + version? I'm not convinced by the latter - it
> would
> >>>> mean some .bom files would work and others wouldn't (e.g. if the .bom
> had
> >>>> multiple items with different versions). Better to support the
> explicit
> >>>> approach IMO.
> >>>>>> ---
> >>>>>> For E ("have a mechanism whereby deployed entities based on an
> affected
> >>>> blueprint are optionally migrated to the new code"), that feels like a
> >>>> separate discussion. It could equally apply to a pure YAML .bom file
> that
> >>>> has been added to the catalog.
> >>>>>> I suggest we discuss that in a separate email thread.
> >>>>>>
> >>>>>> ---
> >>>>>> For (G), it's an interesting suggestion from Svet to make use of
> Karaf
> >>>> Cellar for HA nodes. I'm hesitant (e.g. if restarting a standalone
> >>> Brooklyn
> >>>> node whose VM has died, then it adds big additional requirements for
> what
> >>>> constitutes the "persisted state"). On the other hand, it's good to
> use
> >>>> well-established technologies rather than re-inventing things!
> >>>>>> An alternative ("pure brooklyn") approach could be to write the
> bundle
> >>>> to persisted state; on rebind, we'd install + activate those bundles.
> >>>>>> ---
> >>>>>> For "catalogGroupId", I agree with Svet that in the initial use-case
> >>>> this can be an implementation detail.
> >>>>>> It could be set as the bundle's symbolic name + version: everything
> >>> from
> >>>> the bundle should be deleted at once, along with the bundle.
> >>>>>> Longer term, I can see how exposing "catalogGroupId" to the user
> could
> >>>> support more use-cases (e.g. for several catalog items from different
> >>>> bundles to work together). I don't think we should try to support that
> >>> yet.
> >>>>>> Aled
> >>>>>>
> >>>>>>
> >>>>>> On 19/12/2016 17:19, Geoff Macartney wrote:
> >>>>>>> hi Alex,
> >>>>>>>
> >>>>>>> this looks like a good feature to have, I shall look at the PR as
> soon
> >>>> as I
> >>>>>>> can.
> >>>>>>>
> >>>>>>> The catalog.bom scanner feature was initially enabled by default,
> but
> >>> we
> >>>>>>> had to
> >>>>>>> disable it because it turned out not to work properly with rebind.
> I
> >>>> don't
> >>>>>>> think
> >>>>>>> it should be a lot of work to fix that but it hasn't been something
> >>>> we've
> >>>>>>> got round
> >>>>>>> to yet.  This would be a great opportunity to look back at that.
> >>>>>>>
> >>>>>>> Some random thoughts:
> >>>>>>>
> >>>>>>> re (C), if we are going to treat the zips as bundles, my gut feel
> is
> >>>> that
> >>>>>>> we
> >>>>>>> should insist on a manifest and get the metadata from it.  It
> doesn't
> >>>> feel
> >>>>>>> to me
> >>>>>>> like it makes much sense to allow a zip file without a MANIFEST.MF
> but
> >>>>>>> convey
> >>>>>>> the intended bundle metadata to Brooklyn via HTTP headers.  And
> rather
> >>>> than
> >>>>>>> infer bundle metadata I think it's better to ask users to be
> explicit
> >>>> about
> >>>>>>> what
> >>>>>>> their intentions are.  To make users lives easier, we could
> >>>>>>> add a command to br to generate the manifest (locally) with correct
> >>>> syntax,
> >>>>>>> so that the manifest is in the right place, rather than have br add
> >>> the
> >>>> data
> >>>>>>> to the "upload" request headers.
> >>>>>>>
> >>>>>>> re. (D) will be glad to have a look at it
> >>>>>>>
> >>>>>>> re. (E) it would certainly need to be optional - maybe keep it as
> an
> >>>>>>> explicit
> >>>>>>> separate command ('upgrade'?)
> >>>>>>>
> >>>>>>> (F) it does seem like a lot of work but might be nice for users who
> >>> are
> >>>> not
> >>>>>>> keen on command lines.
> >>>>>>>
> >>>>>>> G - I:  we'll definitely need to pay close attention to persistence
> >>> and
> >>>>>>> rebind;
> >>>>>>> I wonder also about HA operation, are there any additional
> >>> implications?
> >>>>>>> (J) I think it would be good to treat all the files from a jar,
> sorry
> >>>>>>> bundle,
> >>>>>>> as an atomic group - cleaner that way perhaps than allowing
> >>>> delete/update
> >>>>>>> of
> >>>>>>> individual entries from a bundle on a piecemeal basis.  Rest
> support
> >>> on
> >>>>>>> delete
> >>>>>>> catalog could warn about related catalog entries being deleted and
> ask
> >>>> for
> >>>>>>> a "--force" param to confirm.
> >>>>>>>
> >>>>>>> Geoff
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Fri, 16 Dec 2016 at 15:24 Svetoslav Neykov <
> >>>>>>> svetoslav.neykov@cloudsoftcorp.com> wrote:
> >>>>>>>
> >>>>>>>> +1
> >>>>>>>>
> >>>>>>>> Some thoughts:
> >>>>>>>>     * (A) add a utility class BundleMaker
> >>>>>>>>       Sounds very similar to
> >>>>>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles <
> >>>>>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles>
> >>>>>>>>       Looking at the code it's much more focused on zip files so I
> >>> guess
> >>>>>>>> there's no much overlap, but worth keeping in mind
> >>>>>>>>     * (C) accept bundle symbolic name and version
> >>>>>>>>       Why require them at all? Could infer them from the
> catalog.bom
> >>> in
> >>>> some
> >>>>>>>> way - maybe require those properties to be in there. If not
> present
> >>> are
> >>>>>>>> they really needed?
> >>>>>>>>     * (G) Bundles installed via this mechanism are not persisted
> >>>> currently &
> >>>>>>>> (I) We persist the individual catalog items as YAML, so we end up
> >>> with
> >>>> two
> >>>>>>>> records
> >>>>>>>>       Suggest marking the catalog items coming from bundles as
> >>>>>>>> non-persistable. Then try to share the bundles between HA nodes.
> >>> (Karaf
> >>>>>>>> Cellar?)
> >>>>>>>>     * (J) Introduce a catalogGroupId field on catalog items;
> >>>>>>>>       Agree this could be useful and I like the idea of deleting
> the
> >>>> bundle
> >>>>>>>> altogether with the catalog items. From user's perspective I don't
> >>> see
> >>>> the
> >>>>>>>> need for an extra field (i.e. it's an implementation detail).
> >>>>>>>>
> >>>>>>>> Svet.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On 16.12.2016 г., at 12:50, Alex Heneveld <
> >>>>>>>> alex.heneveld@cloudsoftcorp.com> wrote:
> >>>>>>>>> Hi Brooklyners-
> >>>>>>>>>
> >>>>>>>>> In the code we currently have two routes for users to install new
> >>>>>>>>> blueprints:
> >>>>>>>>>
> >>>>>>>>> (1) upload a catalog YAML file to /v1/catalog
> >>>>>>>>>
> >>>>>>>>> (2) install a bundle with catalog.bom in the root
> >>>>>>>>>
> >>>>>>>>> The feature (2) is disabled by default, but I'd like to move
> towards
> >>>>>>>>> enabling it.  This will make it easier to create nicely
> structured
> >>> BOM
> >>>>>>>>> files because scripts etc can be taken out of the BOM, stored as
> >>>> files in
> >>>>>>>>> the same bundle.  (Because URLs of the form
> >>>>>>>>> `classpath://scripts/install.sh`  use the bundle's classpath to
> >>>> resolve.)
> >>>>>>>>> As a first step in #485 [1] I do a few things:
> >>>>>>>>>
> >>>>>>>>> (A) add a utility class BundleMaker that lets us create and
> modify
> >>>>>>>>> bundles/zips, to make it easier to do things we might want to
> with
> >>>>>>>> bundles,
> >>>>>>>>> especially for testing
> >>>>>>>>>
> >>>>>>>>> (B) add an endpoint to the REST API which allows uploading a
> bundle
> >>>> ZIP
> >>>>>>>>> (C) accept bundle symbolic name and version in that REST API to
> >>>>>>>> facilitate
> >>>>>>>>> uploading non-bundle ZIPs where the OSGi MANIFEST.MF is
> >>> automatically
> >>>>>>>>> generated
> >>>>>>>>>
> >>>>>>>>> With this PR, if you have a directory on your local file system
> with
> >>>>>>>>> scripts and config files, and a BOM which refers to them, you can
> >>> just
> >>>>>>>> ZIP
> >>>>>>>>> that up an upload it, specifying the bundle name so that a YAML
> >>>> blueprint
> >>>>>>>>> author never needs to touch any java-isms.
> >>>>>>>>>
> >>>>>>>>> Where I see this going is a development workflow where a user can
> >>> edit
> >>>>>>>>> files locally and upload the ZIP to have that installed, and if
> they
> >>>> make
> >>>>>>>>> changes locally they can POST it again to have catalog items
> updated
> >>>>>>>>> (because default version is a SNAPSHOT).  We could also:
> >>>>>>>>>
> >>>>>>>>> (D) have `br catalog add ~/my/project/ --name my.project` create
> a
> >>> ZIP
> >>>>>>>> and
> >>>>>>>>> POST it, with bundle name metadata, so essentially the user's
> >>> process
> >>>> is
> >>>>>>>>> just to run that whenever they make a change
> >>>>>>>>>
> >>>>>>>>> (E) have a mechanism whereby deployed entities based on an
> affected
> >>>>>>>>> blueprint are optionally migrated to the new code, so if you've
> >>>> changed
> >>>>>>>> an
> >>>>>>>>> enricher the changes are picked up, or if say a launch.sh script
> has
> >>>>>>>>> changed, a restart will run the new code
> >>>>>>>>>
> >>>>>>>>> The above are fairly straightforward programmatically (although
> good
> >>>> user
> >>>>>>>>> interaction with (E) needs some thought).  So I think we can
> pretty
> >>>>>>>> quickly
> >>>>>>>>> get to a much smoother dev workflow.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> That's the highlight of this message.  You can jump to the end,
> >>> unless
> >>>>>>>>> you're interested in some important but low level details...
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> I'm also tempted by:
> >>>>>>>>>
> >>>>>>>>> (F) Integration with web-based IDE and/or Brooklyn reading and
> >>> writing
> >>>>>>>>> straight from GitHub -- but this seems like a lot of work and I'm
> >>> not
> >>>>>>>>> convinced it's much better than (D) workflow-wise
> >>>>>>>>>
> >>>>>>>>> Before we can change (2) to be the default, or start widely using
> >>> the
> >>>>>>>> POST
> >>>>>>>>> a ZIP feature, we need to sort out some issues to do with
> >>> persistence
> >>>> and
> >>>>>>>>> reloading:
> >>>>>>>>>
> >>>>>>>>> (G) Bundles installed via this mechanism are not persisted
> >>> currently,
> >>>> so
> >>>>>>>> if
> >>>>>>>>> you move to a different Brooklyn using the same backing store,
> >>> you'll
> >>>>>>>> lose
> >>>>>>>>> those bundles
> >>>>>>>>>
> >>>>>>>>> (H) On rebind, bundles aren't always activated when needed,
> meaning
> >>>> items
> >>>>>>>>> can't be loaded
> >>>>>>>>>
> >>>>>>>>> (I) We persist the individual catalog items as YAML, so we end up
> >>> with
> >>>>>>>> two
> >>>>>>>>> records — the YAML from the catalog.bom in the bundle, and the
> YAML
> >>>>>>>>> persisted for the item.  This isn't a problem per se, but
> something
> >>> to
> >>>>>>>>> think about, and some sometimes surprising behaviour.  In
> particular
> >>>> if
> >>>>>>>> you
> >>>>>>>>> delete the persisted YAML, the bundle is still there so the item
> is
> >>> no
> >>>>>>>>> longer deleted after a full rebind.
> >>>>>>>>>
> >>>>>>>>> One idea which might be useful is:
> >>>>>>>>>
> >>>>>>>>> (J) Introduce a catalogGroupId field on catalog items; this will
> do
> >>>> two
> >>>>>>>>> things:  if you try to delete an item with such a record, you'll
> be
> >>>>>>>>> encouraged to delete all such items (maybe disallowed to delete
> an
> >>>>>>>>> individual one), with the effect of deleting the bundle if it
> comes
> >>>> from
> >>>>>>>> a
> >>>>>>>>> bundle; and when resolving types we search first for items with
> the
> >>>> same
> >>>>>>>>> catalogGroupId (so that e.g. if I install MyCluster:1.0 and
> >>>> MyNode:1.0 in
> >>>>>>>>> the same group, the former can refer simply to "MyNode" but if I
> >>>> install
> >>>>>>>> a
> >>>>>>>>> 2.0 version of that group, the 1.0 cluster still loads the 1.0
> node
> >>> --
> >>>>>>>> this
> >>>>>>>>> has bitten people i the past)
> >>>>>>>>>
> >>>>>>>>> There is a related Brooklyn upgrade problem worth mentioning,
> which
> >>>> the
> >>>>>>>>> above might help with, where:
> >>>>>>>>>
> >>>>>>>>> (K) If I migrate from Brooklyn 10 to 11 when it comes out, I'll
> no
> >>>> longer
> >>>>>>>>> have certain entities that were at v10, since we don't include
> >>> those;
> >>>> an
> >>>>>>>>> upgrade could include rules that certain groupIds need to be
> >>> updated,
> >>>> or
> >>>>>>>> it
> >>>>>>>>> can search and attempt to automatically apply the updates
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Quite a lot here and we don't need to solve it but I wanted to:
> >>>>>>>>>
> >>>>>>>>> * Share the current thinking
> >>>>>>>>>
> >>>>>>>>> * Get opinions on the general dev workflow suggested by (D)
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Thanks for feedback -- and if we like it help with (D) would be
> >>>>>>>> appreciated!
> >>>>>>>>> Best
> >>>>>>>>> Alex
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> [1] . https://github.com/apache/brooklyn-server/pull/485
> >>>
> >
>
> --
>
> Thomas Bouron • Software Engineer @ Cloudsoft Corporation •
> http://www.cloudsoftcorp.com/
> Github: https://github.com/tbouron
> Twitter: https://twitter.com/eltibouron
>
> --
>
> Thomas Bouron • Software Engineer @ Cloudsoft Corporation •
> http://www.cloudsoftcorp.com/
> Github: https://github.com/tbouron
> Twitter: https://twitter.com/eltibouron
>

Re: Uploading ZIPs for a better dev workflow

[Thomas Bouron <th...@cloudsoftcorp.com>]

Hi all.

So I spent time to review and test [1] and here are the few issues I bumped
into:

   1. the use of `symbolicName` for the bundle clashes Brooklyn: the
   `BasicBrooklynCatalog` watches this key to define catalog items IDs [2]
   2. because of 1. the bundle that I'm trying to install fails. But upon
   failure, the bundle is not uninstalled so any subsequent bundle addition
   fail with the first error
   3. we now have multiple API definitions for the same endpoint (i.e.
   /v1/catalog for yaml and ZIP/JAR) While this is perfectly valid, Swagger
   does not support it [3]. It means that the Swagger UI displays only the
   latest API definition registered, in this case, that's the one to upload
   ZIP/JAR.

For 1. I can see 2 solutions: using a special section `manifest` where we
put the `symbolicName` and `version`, or changing the field name to
`bundle` (I prefer option 2)
For 2. we need to add some logic to uninstall the bundle on failure into
the `CatalogPopulator`, as per as Geoff comment[4]
For 3. even the new Swagger spec (called now OpenAPI) does not support
multiple definitions on the same endpoint [5]. I'm not sure if we can leave
with that. One solution would be to split the endpoint to create specific
one, based on the type of upload.

[1] https://github.com/apache/brooklyn-server/pull/485
[2]
https://github.com/apache/brooklyn-server/blob/master/core/src/main/java/org/apache/brooklyn/core/catalog/internal/BasicBrooklynCatalog.java#L542
[3] https://github.com/swagger-api/swagger-core/issues/935
[4] https://github.com/apache/brooklyn-server/pull/485#discussion_r107482279
[5] https://github.com/OAI/OpenAPI-Specification/issues/182

On Mon, 20 Mar 2017 at 15:16 Thomas Bouron <th...@cloudsoftcorp.com>
wrote:

Hi Brooklyners.

Uploading ZIPs is really something I'm excited to see coming to life so I
would like to help to push this forward.

I've re-read this email thread and it sounds like we have a consensus on
the "one true way" approach, i.e. always requiring the `symbolicName` and
`version` when uploading ZIP/JAR (that includes Svet in his previous email).

One thing not explicitly settled is the format of bom for those 2 fields:
Alex prefers to put these at the root, Aled in a `manifest` section. I
don't have any strong opinion on this: I was leaning toward Aled's
suggestion but the argument that says this applies to the whole bundle make
sense so I now think putting this at the root is probably better. I can see
that Alex already updated his PR[1] to this structure. What the people
think about this?

I'm going to review [1] but are we happy with this consensus? One thing
that will need to be addressed is the persistence/rebind issues but this
can be tackled in following PRs.

Best.

[1] https://github.com/apache/brooklyn-server/pull/485

On Tue, 7 Mar 2017 at 16:18 Svetoslav Neykov <
svetoslav.neykov@cloudsoftcorp.com> wrote:

Having quickly scanned the thread I still don't see a reason why the
requirement to have a consistent naming for the bundle itself which is not
user visible? Could instead have behaviour similar to the machinNamer here.

Slight preference for optional symbolicName when it's a jar file. Though I
really like Geoff's idea of "one true way". The workflow to create a jar
file is very different from creating a zip file (either from a folder or
user supplied) so don't think they conflict.

Svet.


> On 7.03.2017 г., at 17:47, Alex Heneveld <al...@cloudsoftcorp.com>
wrote:
>
>
> Considering the points made I'm pretty sure it's the case that requiring
a symbolic name and version in the BOM presents little if any unnecessary
burden, in which case I'm persuaded of the "one true way" philosophy.  I
really like it.  :)
>
> Can anyone think of a case where it would be a burden?  My thinking is
that:
>
> * If you're uploading a ZIP of YAML and scripts and icons, you have a
catalog.bom and that's a useful place to require the symname and version
> * If your ZIP/JAR also has a MANIFEST.MF the symname and version will be
present in it also; the user is duplicating that info (this is the "little"
unnecssary burden) but it's tiny and the benefit of one true way dominates;
we do require that they match
> * If your ZIP/JAR doesn't have a BOM there's no point in uploading it to
the catalog endpoint; either it's a JAR but this isn't meant as an "install
arbitrary bundles" endpoint, or a non-JAR in which case what are we
supposed to do with it once you upload it?
>
> It keeps the contract simple, and the CLI very simple.   `br push
/path/to/blueprint/x/`
> (But Geoff as a side point I think the CLI should _not_ attempt
validation of the input/version -- the server will do that anyway and no
benefit in doing it twice, but more code to maintain.)
>
> I'm further persuaded given it is in a file, we should require a
version.  I was concerned about needing to supply the version every time on
the CLI but this has gone away.
>
> I see no reason for a separate "manifest" section.  Removing this
eliminates the prospect of Aled's pathological/"perverted" example.  I
favour:
>
>    brooklyn.catalog:
>       symbolicName: com.example.Foo
>       version: 1.0
>       items:
>       - id: foo
>         version: 3.0   # optional version override
>         itemType: entity
>         item:
>           type: blah
>           ...
>
> That is, the fields symbolic-name and version are _required_ under the
`brooklyn.catalog` when POSTing a ZIP to the catalog endpoint, and are used
to define the bundle.  The version (and icon, etc) are then inherited by
default by items defined therein.  If the ZIP is a JAR which contains a
MANIFEST.MF, the bundle information (symbolic name and version) must match
exactly.
>
> Sound right?
>
> Best
> Alex
>
>
> On 03/03/2017 09:59, Geoff Macartney wrote:
>> It also means clients ('br' at least) can easily validate the upload even
>> before execution by checking that all the manifest details are present.
>>
>> On Fri, 3 Mar 2017 at 09:57 Geoff Macartney <
>> geoff.macartney@cloudsoftcorp.com> wrote:
>>
>>> Just on the last point of "We could also support the example below", I'd
>>> say let's not even be that flexible - my feeling is that the more
flexible
>>> we are, the more confusing it is, and the harder to get right.  If we're
>>> going to add this capability let's just have one way to do it; if you
are
>>> going to do a ZIP upload, you MUST have a catalog.bom, which MUST
contain a
>>> manifest section, which MUST have both a symbolic name and a version.
That
>>> way everything's explicit and very clear, every time.
>>>
>>>
>>>
>>> On Thu, 2 Mar 2017 at 11:45 Aled Sage <al...@gmail.com> wrote:
>>>
>>> Geoff, all,
>>>
>>> I was imagining the manifest version (in the catalog.bom) to be separate
>>> from the item version. The reason is that we support multiple items in
>>> the bom that can be independently versioned.
>>>
>>> Somone perverted could write:
>>>
>>>     brooklyn.catalog:
>>>        version: 1.0
>>>        manifest:
>>>          symbolic_name: com.example.Foo
>>>          version: 2.0
>>>        items:
>>>        - id: foo
>>>          version: 3.0
>>>          itemType: entity
>>>          item:
>>>            type: blah
>>>        - id: bar
>>>          version: 4.0
>>>          itemType: entity
>>>          item:
>>>         type: blah.blah
>>>
>>> Here, the "1.0" version is not used by anything; the auto-generated OSGi
>>> bundle would be version 2.0; catalog item foo would be 3.0; and catalog
>>> item bar would be 4.0.
>>>
>>> If we were starting from scratch, I'd be tempted to lock things down a
>>> lot more for what we actually support. For now, it feels like asking for
>>> an explicit version is a reasonable thing to do.
>>>
>>> ---
>>> We could also support the example below, where "1.0" is used by both the
>>> manifest and the item:
>>>
>>>     brooklyn.catalog:
>>>        version: 1.0
>>>        manifest:
>>>          symbolic_name: com.example.Foo
>>>        items:
>>>        - id: foo
>>>          itemType: entity
>>>          item:
>>>            type: blah
>>>
>>> The general rule would be: if there is a version in the manifest
>>> section, that will be used; if not then a top-level version number will
>>> be used. If that is also missing, then fail.
>>>
>>> i.e. we would not accept:
>>>
>>>     # Fails - manifest has no version.
>>>     brooklyn.catalog:
>>>        manifest:
>>>          symbolic_name: com.example.Foo
>>>        items:
>>>        - id: foo
>>>          version: 1.0
>>>          itemType: entity
>>>          item:
>>>            type: blah
>>>
>>> Thoughts?
>>>
>>> Aled
>>>
>>>
>>> On 02/03/2017 10:02, Geoff Macartney wrote:
>>>> I take Alex's point about the manifest being Java specific, and I agree
>>>> therefore we shouldn't insist on it.
>>>>
>>>> +1 also to preferring explicit name/version in the catalog.bom rather
>>> than
>>>> as API params, I agree we do
>>>> need to keep the version in source control.
>>>>
>>>> Question on your straw man, does the 'version' below
>>>>
>>>>      brooklyn.catalog:
>>>>         manifest:
>>>>           symbolic_name: com.example.Foo
>>>>           version: 1.0.0-SNAPSHOT
>>>>         items:
>>>>
>>>> _replace_ the 'version' in existing catalogs
>>>>
>>>> brooklyn.catalog:
>>>>      version: "0.1.0-SNAPSHOT"
>>>>
>>>> or is it an independently variable value, i.e. the version of the
bundle,
>>>> separate from the version of catalog items that it happens to contain?
>>>> (Sounds a bit disquieting, but I guess it's possible).
>>>>
>>>> e.g. could you have
>>>>
>>>> brooklyn.catalog:
>>>>    version: 0.11.0-SNAPSHOT
>>>>    manifest:
>>>>      symbolic_name: com.example.Foo
>>>>      version: 1.0.0-SNAPSHOT
>>>>    items:
>>>>    - id: foo
>>>>      type: xyz
>>>>
>>>> which would be bundle com.example.Foo:1.0.0-SNAPSHOT, which happens to
>>>> contain item foo:0.11.0-SNAPSHOT?
>>>>
>>>> If so, what would happen if you left out the second line above? What
>>>> version of 'foo' would the catalog contain?
>>>>
>>>> Or does the version within the manifest _mean_ that it is not only the
>>>> bundle version you are specifying, but the catalog item versions too?
(I
>>>> guess unless the item explicitly supplies its own version.)
>>>>
>>>> Geoff
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, 1 Mar 2017 at 17:26 Aled Sage <al...@gmail.com> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> Discussion of the REST api kicked off again in
>>>>
>>>
https://github.com/apache/brooklyn-server/pull/485#issuecomment-283280366:
>>>>      Alex wrote:
>>>>
>>>>      Requiring a MANIFEST.MF makes the input strongly java-centric; I'd
>>>>      like to appeal to people who write YAML blueprints with co-bundled
>>>>      scripts and images. They'd wonder why they can't simply make and
>>>>      upload a ZIP. They could probably be persuaded to supply a name
and
>>>>      optional version on a CLI or UI, and understand why it is needed
>>>>      (hence supporting those args) but it would not be idiomatic to
>>>>      anyone but a java programmer to create a META-INF/ dir with a
>>>>      MANIFEST.MF and its syntax. Meanwhile it is very easy for us to
>>>>      convert the ZIP to a JAR on the server. Feels like uncontroversial
>>>>      good UX.
>>>>
>>>>      OTOH for a java programmer a MANIFEST.MF is natural, and they'd
want
>>>>      to drop the name/version args if they are in that file, and I see
no
>>>>      reason to forbid that pattern.
>>>>
>>>> I agree with Alex, that we should not require a META-INF/MANIFEST.MF.
As
>>>> for Geoff's suggestion that we could auto-generate a manifest in the
>>>> `br` CLI, I'd prefer a more general solution that works for users of
the
>>>> REST api as well (i.e. doing it server-side).
>>>>
>>>> ---
>>>> Svet suggested that the catalog.bom could give the symbolic name +
>>>> version, via additional metadata in that file.
>>>>
>>>> What I really like about that is it's in version control, becuase it's
>>>> in the file/repo. If alternatively the name/version are just passed as
>>>> REST api parameters, then it's not in version control (and is more
>>>> susceptible to typos etc).
>>>>
>>>> I'm not sure what we'd want this to look like. As a straw man:
>>>>
>>>>      brooklyn.catalog:
>>>>         manifest:
>>>>           symbolic_name: com.example.Foo
>>>>           version: 1.0.0-SNAPSHOT
>>>>         items:
>>>>         - ...
>>>>
>>>> If the user built their own real OSGi bundle, then they wouldn't need
to
>>>> include this "manifest" section. If they did include that and it
>>>> contradicted the OSGi bundle's manifest, then we'd fail-fast.
>>>>
>>>> (Note this reminds me of the (unrelated) metadata described in
>>>> https://github.com/brooklyncentral/blueprint-repository- there is a
>>>> "publish" block that can be added to the catalog.bom.)
>>>>
>>>> ---
>>>> With Svet's suggestion, if there was no manifest section/file, then
>>>> should we auto-generate something from the item(s) in the catalog.bom?
>>>>
>>>> I can see how that could easily work for a .bom file that has a single
>>>> item (e.g. we use the catalog item's id + version, possibly with a
>>>> special prefix in the symbolic name to avoid conflicts).
>>>>
>>>> However, if there are multiple items then it would get trickier.
>>>>
>>>> I'm inclined to say that for a minimal viable product (MVP) we can
>>>> insist on the "manifest" section in the catalog.bom.
>>>>
>>>> Aled
>>>>
>>>>
>>>> On 20/12/2016 16:34, Svetoslav Neykov wrote:
>>>>>> Svet, if instead we tried to infer it from the catalog.bom, would we
>>>> require some additional metadata within the .bom file? Or would we use
>>> the
>>>> catalog item's id + version? I'm not convinced by the latter - it would
>>>> mean some .bom files would work and others wouldn't (e.g. if the .bom
had
>>>> multiple items with different versions). Better to support the explicit
>>>> approach IMO.
>>>>> I imagine it would be additional metadata. On the other hand I don't
see
>>>> a technical reason why we need an explicit symbolicName and version -
>>> they
>>>> can be auto-generated.
>>>>> Svet.
>>>>>
>>>>>
>>>>>> On 20.12.2016 г., at 17:50, Aled Sage <al...@gmail.com> wrote:
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> +1
>>>>>>
>>>>>> (D) sounds good. What version are you imagining the bundle would be,
if
>>>> one runs `br catalog add ~/my/project/ --name com.example.myproject`?
>>>>>> ---
>>>>>> I like the idea of uploading a plain zip (rather than only supporting
>>>> OSGi bundles) - that makes it simpler for non-java folk. The use of
OSGi
>>>> becomes a (hidden) implementation detail to many users.
>>>>>> ---
>>>>>> If auto-generating the manifest, I think we need the user to be
>>> explicit
>>>> about symbolic name and version. Having these supplied in the REST api
>>> call
>>>> (as Alex suggests) would achieve that.
>>>>>> Svet, if instead we tried to infer it from the catalog.bom, would we
>>>> require some additional metadata within the .bom file? Or would we use
>>> the
>>>> catalog item's id + version? I'm not convinced by the latter - it would
>>>> mean some .bom files would work and others wouldn't (e.g. if the .bom
had
>>>> multiple items with different versions). Better to support the explicit
>>>> approach IMO.
>>>>>> ---
>>>>>> For E ("have a mechanism whereby deployed entities based on an
affected
>>>> blueprint are optionally migrated to the new code"), that feels like a
>>>> separate discussion. It could equally apply to a pure YAML .bom file
that
>>>> has been added to the catalog.
>>>>>> I suggest we discuss that in a separate email thread.
>>>>>>
>>>>>> ---
>>>>>> For (G), it's an interesting suggestion from Svet to make use of
Karaf
>>>> Cellar for HA nodes. I'm hesitant (e.g. if restarting a standalone
>>> Brooklyn
>>>> node whose VM has died, then it adds big additional requirements for
what
>>>> constitutes the "persisted state"). On the other hand, it's good to use
>>>> well-established technologies rather than re-inventing things!
>>>>>> An alternative ("pure brooklyn") approach could be to write the
bundle
>>>> to persisted state; on rebind, we'd install + activate those bundles.
>>>>>> ---
>>>>>> For "catalogGroupId", I agree with Svet that in the initial use-case
>>>> this can be an implementation detail.
>>>>>> It could be set as the bundle's symbolic name + version: everything
>>> from
>>>> the bundle should be deleted at once, along with the bundle.
>>>>>> Longer term, I can see how exposing "catalogGroupId" to the user
could
>>>> support more use-cases (e.g. for several catalog items from different
>>>> bundles to work together). I don't think we should try to support that
>>> yet.
>>>>>> Aled
>>>>>>
>>>>>>
>>>>>> On 19/12/2016 17:19, Geoff Macartney wrote:
>>>>>>> hi Alex,
>>>>>>>
>>>>>>> this looks like a good feature to have, I shall look at the PR as
soon
>>>> as I
>>>>>>> can.
>>>>>>>
>>>>>>> The catalog.bom scanner feature was initially enabled by default,
but
>>> we
>>>>>>> had to
>>>>>>> disable it because it turned out not to work properly with rebind. I
>>>> don't
>>>>>>> think
>>>>>>> it should be a lot of work to fix that but it hasn't been something
>>>> we've
>>>>>>> got round
>>>>>>> to yet.  This would be a great opportunity to look back at that.
>>>>>>>
>>>>>>> Some random thoughts:
>>>>>>>
>>>>>>> re (C), if we are going to treat the zips as bundles, my gut feel is
>>>> that
>>>>>>> we
>>>>>>> should insist on a manifest and get the metadata from it.  It
doesn't
>>>> feel
>>>>>>> to me
>>>>>>> like it makes much sense to allow a zip file without a MANIFEST.MF
but
>>>>>>> convey
>>>>>>> the intended bundle metadata to Brooklyn via HTTP headers.  And
rather
>>>> than
>>>>>>> infer bundle metadata I think it's better to ask users to be
explicit
>>>> about
>>>>>>> what
>>>>>>> their intentions are.  To make users lives easier, we could
>>>>>>> add a command to br to generate the manifest (locally) with correct
>>>> syntax,
>>>>>>> so that the manifest is in the right place, rather than have br add
>>> the
>>>> data
>>>>>>> to the "upload" request headers.
>>>>>>>
>>>>>>> re. (D) will be glad to have a look at it
>>>>>>>
>>>>>>> re. (E) it would certainly need to be optional - maybe keep it as an
>>>>>>> explicit
>>>>>>> separate command ('upgrade'?)
>>>>>>>
>>>>>>> (F) it does seem like a lot of work but might be nice for users who
>>> are
>>>> not
>>>>>>> keen on command lines.
>>>>>>>
>>>>>>> G - I:  we'll definitely need to pay close attention to persistence
>>> and
>>>>>>> rebind;
>>>>>>> I wonder also about HA operation, are there any additional
>>> implications?
>>>>>>> (J) I think it would be good to treat all the files from a jar,
sorry
>>>>>>> bundle,
>>>>>>> as an atomic group - cleaner that way perhaps than allowing
>>>> delete/update
>>>>>>> of
>>>>>>> individual entries from a bundle on a piecemeal basis.  Rest support
>>> on
>>>>>>> delete
>>>>>>> catalog could warn about related catalog entries being deleted and
ask
>>>> for
>>>>>>> a "--force" param to confirm.
>>>>>>>
>>>>>>> Geoff
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, 16 Dec 2016 at 15:24 Svetoslav Neykov <
>>>>>>> svetoslav.neykov@cloudsoftcorp.com> wrote:
>>>>>>>
>>>>>>>> +1
>>>>>>>>
>>>>>>>> Some thoughts:
>>>>>>>>     * (A) add a utility class BundleMaker
>>>>>>>>       Sounds very similar to
>>>>>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles <
>>>>>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles>
>>>>>>>>       Looking at the code it's much more focused on zip files so I
>>> guess
>>>>>>>> there's no much overlap, but worth keeping in mind
>>>>>>>>     * (C) accept bundle symbolic name and version
>>>>>>>>       Why require them at all? Could infer them from the
catalog.bom
>>> in
>>>> some
>>>>>>>> way - maybe require those properties to be in there. If not present
>>> are
>>>>>>>> they really needed?
>>>>>>>>     * (G) Bundles installed via this mechanism are not persisted
>>>> currently &
>>>>>>>> (I) We persist the individual catalog items as YAML, so we end up
>>> with
>>>> two
>>>>>>>> records
>>>>>>>>       Suggest marking the catalog items coming from bundles as
>>>>>>>> non-persistable. Then try to share the bundles between HA nodes.
>>> (Karaf
>>>>>>>> Cellar?)
>>>>>>>>     * (J) Introduce a catalogGroupId field on catalog items;
>>>>>>>>       Agree this could be useful and I like the idea of deleting
the
>>>> bundle
>>>>>>>> altogether with the catalog items. From user's perspective I don't
>>> see
>>>> the
>>>>>>>> need for an extra field (i.e. it's an implementation detail).
>>>>>>>>
>>>>>>>> Svet.
>>>>>>>>
>>>>>>>>
>>>>>>>>> On 16.12.2016 г., at 12:50, Alex Heneveld <
>>>>>>>> alex.heneveld@cloudsoftcorp.com> wrote:
>>>>>>>>> Hi Brooklyners-
>>>>>>>>>
>>>>>>>>> In the code we currently have two routes for users to install new
>>>>>>>>> blueprints:
>>>>>>>>>
>>>>>>>>> (1) upload a catalog YAML file to /v1/catalog
>>>>>>>>>
>>>>>>>>> (2) install a bundle with catalog.bom in the root
>>>>>>>>>
>>>>>>>>> The feature (2) is disabled by default, but I'd like to move
towards
>>>>>>>>> enabling it.  This will make it easier to create nicely structured
>>> BOM
>>>>>>>>> files because scripts etc can be taken out of the BOM, stored as
>>>> files in
>>>>>>>>> the same bundle.  (Because URLs of the form
>>>>>>>>> `classpath://scripts/install.sh`  use the bundle's classpath to
>>>> resolve.)
>>>>>>>>> As a first step in #485 [1] I do a few things:
>>>>>>>>>
>>>>>>>>> (A) add a utility class BundleMaker that lets us create and modify
>>>>>>>>> bundles/zips, to make it easier to do things we might want to with
>>>>>>>> bundles,
>>>>>>>>> especially for testing
>>>>>>>>>
>>>>>>>>> (B) add an endpoint to the REST API which allows uploading a
bundle
>>>> ZIP
>>>>>>>>> (C) accept bundle symbolic name and version in that REST API to
>>>>>>>> facilitate
>>>>>>>>> uploading non-bundle ZIPs where the OSGi MANIFEST.MF is
>>> automatically
>>>>>>>>> generated
>>>>>>>>>
>>>>>>>>> With this PR, if you have a directory on your local file system
with
>>>>>>>>> scripts and config files, and a BOM which refers to them, you can
>>> just
>>>>>>>> ZIP
>>>>>>>>> that up an upload it, specifying the bundle name so that a YAML
>>>> blueprint
>>>>>>>>> author never needs to touch any java-isms.
>>>>>>>>>
>>>>>>>>> Where I see this going is a development workflow where a user can
>>> edit
>>>>>>>>> files locally and upload the ZIP to have that installed, and if
they
>>>> make
>>>>>>>>> changes locally they can POST it again to have catalog items
updated
>>>>>>>>> (because default version is a SNAPSHOT).  We could also:
>>>>>>>>>
>>>>>>>>> (D) have `br catalog add ~/my/project/ --name my.project` create a
>>> ZIP
>>>>>>>> and
>>>>>>>>> POST it, with bundle name metadata, so essentially the user's
>>> process
>>>> is
>>>>>>>>> just to run that whenever they make a change
>>>>>>>>>
>>>>>>>>> (E) have a mechanism whereby deployed entities based on an
affected
>>>>>>>>> blueprint are optionally migrated to the new code, so if you've
>>>> changed
>>>>>>>> an
>>>>>>>>> enricher the changes are picked up, or if say a launch.sh script
has
>>>>>>>>> changed, a restart will run the new code
>>>>>>>>>
>>>>>>>>> The above are fairly straightforward programmatically (although
good
>>>> user
>>>>>>>>> interaction with (E) needs some thought).  So I think we can
pretty
>>>>>>>> quickly
>>>>>>>>> get to a much smoother dev workflow.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> That's the highlight of this message.  You can jump to the end,
>>> unless
>>>>>>>>> you're interested in some important but low level details...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'm also tempted by:
>>>>>>>>>
>>>>>>>>> (F) Integration with web-based IDE and/or Brooklyn reading and
>>> writing
>>>>>>>>> straight from GitHub -- but this seems like a lot of work and I'm
>>> not
>>>>>>>>> convinced it's much better than (D) workflow-wise
>>>>>>>>>
>>>>>>>>> Before we can change (2) to be the default, or start widely using
>>> the
>>>>>>>> POST
>>>>>>>>> a ZIP feature, we need to sort out some issues to do with
>>> persistence
>>>> and
>>>>>>>>> reloading:
>>>>>>>>>
>>>>>>>>> (G) Bundles installed via this mechanism are not persisted
>>> currently,
>>>> so
>>>>>>>> if
>>>>>>>>> you move to a different Brooklyn using the same backing store,
>>> you'll
>>>>>>>> lose
>>>>>>>>> those bundles
>>>>>>>>>
>>>>>>>>> (H) On rebind, bundles aren't always activated when needed,
meaning
>>>> items
>>>>>>>>> can't be loaded
>>>>>>>>>
>>>>>>>>> (I) We persist the individual catalog items as YAML, so we end up
>>> with
>>>>>>>> two
>>>>>>>>> records — the YAML from the catalog.bom in the bundle, and the
YAML
>>>>>>>>> persisted for the item.  This isn't a problem per se, but
something
>>> to
>>>>>>>>> think about, and some sometimes surprising behaviour.  In
particular
>>>> if
>>>>>>>> you
>>>>>>>>> delete the persisted YAML, the bundle is still there so the item
is
>>> no
>>>>>>>>> longer deleted after a full rebind.
>>>>>>>>>
>>>>>>>>> One idea which might be useful is:
>>>>>>>>>
>>>>>>>>> (J) Introduce a catalogGroupId field on catalog items; this will
do
>>>> two
>>>>>>>>> things:  if you try to delete an item with such a record, you'll
be
>>>>>>>>> encouraged to delete all such items (maybe disallowed to delete an
>>>>>>>>> individual one), with the effect of deleting the bundle if it
comes
>>>> from
>>>>>>>> a
>>>>>>>>> bundle; and when resolving types we search first for items with
the
>>>> same
>>>>>>>>> catalogGroupId (so that e.g. if I install MyCluster:1.0 and
>>>> MyNode:1.0 in
>>>>>>>>> the same group, the former can refer simply to "MyNode" but if I
>>>> install
>>>>>>>> a
>>>>>>>>> 2.0 version of that group, the 1.0 cluster still loads the 1.0
node
>>> --
>>>>>>>> this
>>>>>>>>> has bitten people i the past)
>>>>>>>>>
>>>>>>>>> There is a related Brooklyn upgrade problem worth mentioning,
which
>>>> the
>>>>>>>>> above might help with, where:
>>>>>>>>>
>>>>>>>>> (K) If I migrate from Brooklyn 10 to 11 when it comes out, I'll no
>>>> longer
>>>>>>>>> have certain entities that were at v10, since we don't include
>>> those;
>>>> an
>>>>>>>>> upgrade could include rules that certain groupIds need to be
>>> updated,
>>>> or
>>>>>>>> it
>>>>>>>>> can search and attempt to automatically apply the updates
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Quite a lot here and we don't need to solve it but I wanted to:
>>>>>>>>>
>>>>>>>>> * Share the current thinking
>>>>>>>>>
>>>>>>>>> * Get opinions on the general dev workflow suggested by (D)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks for feedback -- and if we like it help with (D) would be
>>>>>>>> appreciated!
>>>>>>>>> Best
>>>>>>>>> Alex
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [1] . https://github.com/apache/brooklyn-server/pull/485
>>>
>

-- 

Thomas Bouron • Software Engineer @ Cloudsoft Corporation •
http://www.cloudsoftcorp.com/
Github: https://github.com/tbouron
Twitter: https://twitter.com/eltibouron

-- 

Thomas Bouron • Software Engineer @ Cloudsoft Corporation •
http://www.cloudsoftcorp.com/
Github: https://github.com/tbouron
Twitter: https://twitter.com/eltibouron

Re: Uploading ZIPs for a better dev workflow

[Thomas Bouron <th...@cloudsoftcorp.com>]

Hi Brooklyners.

Uploading ZIPs is really something I'm excited to see coming to life so I
would like to help to push this forward.

I've re-read this email thread and it sounds like we have a consensus on
the "one true way" approach, i.e. always requiring the `symbolicName` and
`version` when uploading ZIP/JAR (that includes Svet in his previous email).

One thing not explicitly settled is the format of bom for those 2 fields:
Alex prefers to put these at the root, Aled in a `manifest` section. I
don't have any strong opinion on this: I was leaning toward Aled's
suggestion but the argument that says this applies to the whole bundle make
sense so I now think putting this at the root is probably better. I can see
that Alex already updated his PR[1] to this structure. What the people
think about this?

I'm going to review [1] but are we happy with this consensus? One thing
that will need to be addressed is the persistence/rebind issues but this
can be tackled in following PRs.

Best.

[1] https://github.com/apache/brooklyn-server/pull/485

On Tue, 7 Mar 2017 at 16:18 Svetoslav Neykov <
svetoslav.neykov@cloudsoftcorp.com> wrote:

> Having quickly scanned the thread I still don't see a reason why the
> requirement to have a consistent naming for the bundle itself which is not
> user visible? Could instead have behaviour similar to the machinNamer here.
>
> Slight preference for optional symbolicName when it's a jar file. Though I
> really like Geoff's idea of "one true way". The workflow to create a jar
> file is very different from creating a zip file (either from a folder or
> user supplied) so don't think they conflict.
>
> Svet.
>
>
> > On 7.03.2017 г., at 17:47, Alex Heneveld <
> alex.heneveld@cloudsoftcorp.com> wrote:
> >
> >
> > Considering the points made I'm pretty sure it's the case that requiring
> a symbolic name and version in the BOM presents little if any unnecessary
> burden, in which case I'm persuaded of the "one true way" philosophy.  I
> really like it.  :)
> >
> > Can anyone think of a case where it would be a burden?  My thinking is
> that:
> >
> > * If you're uploading a ZIP of YAML and scripts and icons, you have a
> catalog.bom and that's a useful place to require the symname and version
> > * If your ZIP/JAR also has a MANIFEST.MF the symname and version will be
> present in it also; the user is duplicating that info (this is the "little"
> unnecssary burden) but it's tiny and the benefit of one true way dominates;
> we do require that they match
> > * If your ZIP/JAR doesn't have a BOM there's no point in uploading it to
> the catalog endpoint; either it's a JAR but this isn't meant as an "install
> arbitrary bundles" endpoint, or a non-JAR in which case what are we
> supposed to do with it once you upload it?
> >
> > It keeps the contract simple, and the CLI very simple.   `br push
> /path/to/blueprint/x/`
> > (But Geoff as a side point I think the CLI should _not_ attempt
> validation of the input/version -- the server will do that anyway and no
> benefit in doing it twice, but more code to maintain.)
> >
> > I'm further persuaded given it is in a file, we should require a
> version.  I was concerned about needing to supply the version every time on
> the CLI but this has gone away.
> >
> > I see no reason for a separate "manifest" section.  Removing this
> eliminates the prospect of Aled's pathological/"perverted" example.  I
> favour:
> >
> >    brooklyn.catalog:
> >       symbolicName: com.example.Foo
> >       version: 1.0
> >       items:
> >       - id: foo
> >         version: 3.0   # optional version override
> >         itemType: entity
> >         item:
> >           type: blah
> >           ...
> >
> > That is, the fields symbolic-name and version are _required_ under the
> `brooklyn.catalog` when POSTing a ZIP to the catalog endpoint, and are used
> to define the bundle.  The version (and icon, etc) are then inherited by
> default by items defined therein.  If the ZIP is a JAR which contains a
> MANIFEST.MF, the bundle information (symbolic name and version) must match
> exactly.
> >
> > Sound right?
> >
> > Best
> > Alex
> >
> >
> > On 03/03/2017 09:59, Geoff Macartney wrote:
> >> It also means clients ('br' at least) can easily validate the upload
> even
> >> before execution by checking that all the manifest details are present.
> >>
> >> On Fri, 3 Mar 2017 at 09:57 Geoff Macartney <
> >> geoff.macartney@cloudsoftcorp.com> wrote:
> >>
> >>> Just on the last point of "We could also support the example below",
> I'd
> >>> say let's not even be that flexible - my feeling is that the more
> flexible
> >>> we are, the more confusing it is, and the harder to get right.  If
> we're
> >>> going to add this capability let's just have one way to do it; if you
> are
> >>> going to do a ZIP upload, you MUST have a catalog.bom, which MUST
> contain a
> >>> manifest section, which MUST have both a symbolic name and a version.
> That
> >>> way everything's explicit and very clear, every time.
> >>>
> >>>
> >>>
> >>> On Thu, 2 Mar 2017 at 11:45 Aled Sage <al...@gmail.com> wrote:
> >>>
> >>> Geoff, all,
> >>>
> >>> I was imagining the manifest version (in the catalog.bom) to be
> separate
> >>> from the item version. The reason is that we support multiple items in
> >>> the bom that can be independently versioned.
> >>>
> >>> Somone perverted could write:
> >>>
> >>>     brooklyn.catalog:
> >>>        version: 1.0
> >>>        manifest:
> >>>          symbolic_name: com.example.Foo
> >>>          version: 2.0
> >>>        items:
> >>>        - id: foo
> >>>          version: 3.0
> >>>          itemType: entity
> >>>          item:
> >>>            type: blah
> >>>        - id: bar
> >>>          version: 4.0
> >>>          itemType: entity
> >>>          item:
> >>>         type: blah.blah
> >>>
> >>> Here, the "1.0" version is not used by anything; the auto-generated
> OSGi
> >>> bundle would be version 2.0; catalog item foo would be 3.0; and catalog
> >>> item bar would be 4.0.
> >>>
> >>> If we were starting from scratch, I'd be tempted to lock things down a
> >>> lot more for what we actually support. For now, it feels like asking
> for
> >>> an explicit version is a reasonable thing to do.
> >>>
> >>> ---
> >>> We could also support the example below, where "1.0" is used by both
> the
> >>> manifest and the item:
> >>>
> >>>     brooklyn.catalog:
> >>>        version: 1.0
> >>>        manifest:
> >>>          symbolic_name: com.example.Foo
> >>>        items:
> >>>        - id: foo
> >>>          itemType: entity
> >>>          item:
> >>>            type: blah
> >>>
> >>> The general rule would be: if there is a version in the manifest
> >>> section, that will be used; if not then a top-level version number will
> >>> be used. If that is also missing, then fail.
> >>>
> >>> i.e. we would not accept:
> >>>
> >>>     # Fails - manifest has no version.
> >>>     brooklyn.catalog:
> >>>        manifest:
> >>>          symbolic_name: com.example.Foo
> >>>        items:
> >>>        - id: foo
> >>>          version: 1.0
> >>>          itemType: entity
> >>>          item:
> >>>            type: blah
> >>>
> >>> Thoughts?
> >>>
> >>> Aled
> >>>
> >>>
> >>> On 02/03/2017 10:02, Geoff Macartney wrote:
> >>>> I take Alex's point about the manifest being Java specific, and I
> agree
> >>>> therefore we shouldn't insist on it.
> >>>>
> >>>> +1 also to preferring explicit name/version in the catalog.bom rather
> >>> than
> >>>> as API params, I agree we do
> >>>> need to keep the version in source control.
> >>>>
> >>>> Question on your straw man, does the 'version' below
> >>>>
> >>>>      brooklyn.catalog:
> >>>>         manifest:
> >>>>           symbolic_name: com.example.Foo
> >>>>           version: 1.0.0-SNAPSHOT
> >>>>         items:
> >>>>
> >>>> _replace_ the 'version' in existing catalogs
> >>>>
> >>>> brooklyn.catalog:
> >>>>      version: "0.1.0-SNAPSHOT"
> >>>>
> >>>> or is it an independently variable value, i.e. the version of the
> bundle,
> >>>> separate from the version of catalog items that it happens to contain?
> >>>> (Sounds a bit disquieting, but I guess it's possible).
> >>>>
> >>>> e.g. could you have
> >>>>
> >>>> brooklyn.catalog:
> >>>>    version: 0.11.0-SNAPSHOT
> >>>>    manifest:
> >>>>      symbolic_name: com.example.Foo
> >>>>      version: 1.0.0-SNAPSHOT
> >>>>    items:
> >>>>    - id: foo
> >>>>      type: xyz
> >>>>
> >>>> which would be bundle com.example.Foo:1.0.0-SNAPSHOT, which happens to
> >>>> contain item foo:0.11.0-SNAPSHOT?
> >>>>
> >>>> If so, what would happen if you left out the second line above? What
> >>>> version of 'foo' would the catalog contain?
> >>>>
> >>>> Or does the version within the manifest _mean_ that it is not only the
> >>>> bundle version you are specifying, but the catalog item versions too?
> (I
> >>>> guess unless the item explicitly supplies its own version.)
> >>>>
> >>>> Geoff
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Wed, 1 Mar 2017 at 17:26 Aled Sage <al...@gmail.com> wrote:
> >>>>
> >>>> Hi all,
> >>>>
> >>>> Discussion of the REST api kicked off again in
> >>>>
> >>>
> https://github.com/apache/brooklyn-server/pull/485#issuecomment-283280366:
> >>>>      Alex wrote:
> >>>>
> >>>>      Requiring a MANIFEST.MF makes the input strongly java-centric;
> I'd
> >>>>      like to appeal to people who write YAML blueprints with
> co-bundled
> >>>>      scripts and images. They'd wonder why they can't simply make and
> >>>>      upload a ZIP. They could probably be persuaded to supply a name
> and
> >>>>      optional version on a CLI or UI, and understand why it is needed
> >>>>      (hence supporting those args) but it would not be idiomatic to
> >>>>      anyone but a java programmer to create a META-INF/ dir with a
> >>>>      MANIFEST.MF and its syntax. Meanwhile it is very easy for us to
> >>>>      convert the ZIP to a JAR on the server. Feels like
> uncontroversial
> >>>>      good UX.
> >>>>
> >>>>      OTOH for a java programmer a MANIFEST.MF is natural, and they'd
> want
> >>>>      to drop the name/version args if they are in that file, and I
> see no
> >>>>      reason to forbid that pattern.
> >>>>
> >>>> I agree with Alex, that we should not require a META-INF/MANIFEST.MF.
> As
> >>>> for Geoff's suggestion that we could auto-generate a manifest in the
> >>>> `br` CLI, I'd prefer a more general solution that works for users of
> the
> >>>> REST api as well (i.e. doing it server-side).
> >>>>
> >>>> ---
> >>>> Svet suggested that the catalog.bom could give the symbolic name +
> >>>> version, via additional metadata in that file.
> >>>>
> >>>> What I really like about that is it's in version control, becuase it's
> >>>> in the file/repo. If alternatively the name/version are just passed as
> >>>> REST api parameters, then it's not in version control (and is more
> >>>> susceptible to typos etc).
> >>>>
> >>>> I'm not sure what we'd want this to look like. As a straw man:
> >>>>
> >>>>      brooklyn.catalog:
> >>>>         manifest:
> >>>>           symbolic_name: com.example.Foo
> >>>>           version: 1.0.0-SNAPSHOT
> >>>>         items:
> >>>>         - ...
> >>>>
> >>>> If the user built their own real OSGi bundle, then they wouldn't need
> to
> >>>> include this "manifest" section. If they did include that and it
> >>>> contradicted the OSGi bundle's manifest, then we'd fail-fast.
> >>>>
> >>>> (Note this reminds me of the (unrelated) metadata described in
> >>>> https://github.com/brooklyncentral/blueprint-repository- there is a
> >>>> "publish" block that can be added to the catalog.bom.)
> >>>>
> >>>> ---
> >>>> With Svet's suggestion, if there was no manifest section/file, then
> >>>> should we auto-generate something from the item(s) in the catalog.bom?
> >>>>
> >>>> I can see how that could easily work for a .bom file that has a single
> >>>> item (e.g. we use the catalog item's id + version, possibly with a
> >>>> special prefix in the symbolic name to avoid conflicts).
> >>>>
> >>>> However, if there are multiple items then it would get trickier.
> >>>>
> >>>> I'm inclined to say that for a minimal viable product (MVP) we can
> >>>> insist on the "manifest" section in the catalog.bom.
> >>>>
> >>>> Aled
> >>>>
> >>>>
> >>>> On 20/12/2016 16:34, Svetoslav Neykov wrote:
> >>>>>> Svet, if instead we tried to infer it from the catalog.bom, would we
> >>>> require some additional metadata within the .bom file? Or would we use
> >>> the
> >>>> catalog item's id + version? I'm not convinced by the latter - it
> would
> >>>> mean some .bom files would work and others wouldn't (e.g. if the .bom
> had
> >>>> multiple items with different versions). Better to support the
> explicit
> >>>> approach IMO.
> >>>>> I imagine it would be additional metadata. On the other hand I don't
> see
> >>>> a technical reason why we need an explicit symbolicName and version -
> >>> they
> >>>> can be auto-generated.
> >>>>> Svet.
> >>>>>
> >>>>>
> >>>>>> On 20.12.2016 г., at 17:50, Aled Sage <al...@gmail.com> wrote:
> >>>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> +1
> >>>>>>
> >>>>>> (D) sounds good. What version are you imagining the bundle would
> be, if
> >>>> one runs `br catalog add ~/my/project/ --name com.example.myproject`?
> >>>>>> ---
> >>>>>> I like the idea of uploading a plain zip (rather than only
> supporting
> >>>> OSGi bundles) - that makes it simpler for non-java folk. The use of
> OSGi
> >>>> becomes a (hidden) implementation detail to many users.
> >>>>>> ---
> >>>>>> If auto-generating the manifest, I think we need the user to be
> >>> explicit
> >>>> about symbolic name and version. Having these supplied in the REST api
> >>> call
> >>>> (as Alex suggests) would achieve that.
> >>>>>> Svet, if instead we tried to infer it from the catalog.bom, would we
> >>>> require some additional metadata within the .bom file? Or would we use
> >>> the
> >>>> catalog item's id + version? I'm not convinced by the latter - it
> would
> >>>> mean some .bom files would work and others wouldn't (e.g. if the .bom
> had
> >>>> multiple items with different versions). Better to support the
> explicit
> >>>> approach IMO.
> >>>>>> ---
> >>>>>> For E ("have a mechanism whereby deployed entities based on an
> affected
> >>>> blueprint are optionally migrated to the new code"), that feels like a
> >>>> separate discussion. It could equally apply to a pure YAML .bom file
> that
> >>>> has been added to the catalog.
> >>>>>> I suggest we discuss that in a separate email thread.
> >>>>>>
> >>>>>> ---
> >>>>>> For (G), it's an interesting suggestion from Svet to make use of
> Karaf
> >>>> Cellar for HA nodes. I'm hesitant (e.g. if restarting a standalone
> >>> Brooklyn
> >>>> node whose VM has died, then it adds big additional requirements for
> what
> >>>> constitutes the "persisted state"). On the other hand, it's good to
> use
> >>>> well-established technologies rather than re-inventing things!
> >>>>>> An alternative ("pure brooklyn") approach could be to write the
> bundle
> >>>> to persisted state; on rebind, we'd install + activate those bundles.
> >>>>>> ---
> >>>>>> For "catalogGroupId", I agree with Svet that in the initial use-case
> >>>> this can be an implementation detail.
> >>>>>> It could be set as the bundle's symbolic name + version: everything
> >>> from
> >>>> the bundle should be deleted at once, along with the bundle.
> >>>>>> Longer term, I can see how exposing "catalogGroupId" to the user
> could
> >>>> support more use-cases (e.g. for several catalog items from different
> >>>> bundles to work together). I don't think we should try to support that
> >>> yet.
> >>>>>> Aled
> >>>>>>
> >>>>>>
> >>>>>> On 19/12/2016 17:19, Geoff Macartney wrote:
> >>>>>>> hi Alex,
> >>>>>>>
> >>>>>>> this looks like a good feature to have, I shall look at the PR as
> soon
> >>>> as I
> >>>>>>> can.
> >>>>>>>
> >>>>>>> The catalog.bom scanner feature was initially enabled by default,
> but
> >>> we
> >>>>>>> had to
> >>>>>>> disable it because it turned out not to work properly with rebind.
> I
> >>>> don't
> >>>>>>> think
> >>>>>>> it should be a lot of work to fix that but it hasn't been something
> >>>> we've
> >>>>>>> got round
> >>>>>>> to yet.  This would be a great opportunity to look back at that.
> >>>>>>>
> >>>>>>> Some random thoughts:
> >>>>>>>
> >>>>>>> re (C), if we are going to treat the zips as bundles, my gut feel
> is
> >>>> that
> >>>>>>> we
> >>>>>>> should insist on a manifest and get the metadata from it.  It
> doesn't
> >>>> feel
> >>>>>>> to me
> >>>>>>> like it makes much sense to allow a zip file without a MANIFEST.MF
> but
> >>>>>>> convey
> >>>>>>> the intended bundle metadata to Brooklyn via HTTP headers.  And
> rather
> >>>> than
> >>>>>>> infer bundle metadata I think it's better to ask users to be
> explicit
> >>>> about
> >>>>>>> what
> >>>>>>> their intentions are.  To make users lives easier, we could
> >>>>>>> add a command to br to generate the manifest (locally) with correct
> >>>> syntax,
> >>>>>>> so that the manifest is in the right place, rather than have br add
> >>> the
> >>>> data
> >>>>>>> to the "upload" request headers.
> >>>>>>>
> >>>>>>> re. (D) will be glad to have a look at it
> >>>>>>>
> >>>>>>> re. (E) it would certainly need to be optional - maybe keep it as
> an
> >>>>>>> explicit
> >>>>>>> separate command ('upgrade'?)
> >>>>>>>
> >>>>>>> (F) it does seem like a lot of work but might be nice for users who
> >>> are
> >>>> not
> >>>>>>> keen on command lines.
> >>>>>>>
> >>>>>>> G - I:  we'll definitely need to pay close attention to persistence
> >>> and
> >>>>>>> rebind;
> >>>>>>> I wonder also about HA operation, are there any additional
> >>> implications?
> >>>>>>> (J) I think it would be good to treat all the files from a jar,
> sorry
> >>>>>>> bundle,
> >>>>>>> as an atomic group - cleaner that way perhaps than allowing
> >>>> delete/update
> >>>>>>> of
> >>>>>>> individual entries from a bundle on a piecemeal basis.  Rest
> support
> >>> on
> >>>>>>> delete
> >>>>>>> catalog could warn about related catalog entries being deleted and
> ask
> >>>> for
> >>>>>>> a "--force" param to confirm.
> >>>>>>>
> >>>>>>> Geoff
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Fri, 16 Dec 2016 at 15:24 Svetoslav Neykov <
> >>>>>>> svetoslav.neykov@cloudsoftcorp.com> wrote:
> >>>>>>>
> >>>>>>>> +1
> >>>>>>>>
> >>>>>>>> Some thoughts:
> >>>>>>>>     * (A) add a utility class BundleMaker
> >>>>>>>>       Sounds very similar to
> >>>>>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles <
> >>>>>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles>
> >>>>>>>>       Looking at the code it's much more focused on zip files so I
> >>> guess
> >>>>>>>> there's no much overlap, but worth keeping in mind
> >>>>>>>>     * (C) accept bundle symbolic name and version
> >>>>>>>>       Why require them at all? Could infer them from the
> catalog.bom
> >>> in
> >>>> some
> >>>>>>>> way - maybe require those properties to be in there. If not
> present
> >>> are
> >>>>>>>> they really needed?
> >>>>>>>>     * (G) Bundles installed via this mechanism are not persisted
> >>>> currently &
> >>>>>>>> (I) We persist the individual catalog items as YAML, so we end up
> >>> with
> >>>> two
> >>>>>>>> records
> >>>>>>>>       Suggest marking the catalog items coming from bundles as
> >>>>>>>> non-persistable. Then try to share the bundles between HA nodes.
> >>> (Karaf
> >>>>>>>> Cellar?)
> >>>>>>>>     * (J) Introduce a catalogGroupId field on catalog items;
> >>>>>>>>       Agree this could be useful and I like the idea of deleting
> the
> >>>> bundle
> >>>>>>>> altogether with the catalog items. From user's perspective I don't
> >>> see
> >>>> the
> >>>>>>>> need for an extra field (i.e. it's an implementation detail).
> >>>>>>>>
> >>>>>>>> Svet.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On 16.12.2016 г., at 12:50, Alex Heneveld <
> >>>>>>>> alex.heneveld@cloudsoftcorp.com> wrote:
> >>>>>>>>> Hi Brooklyners-
> >>>>>>>>>
> >>>>>>>>> In the code we currently have two routes for users to install new
> >>>>>>>>> blueprints:
> >>>>>>>>>
> >>>>>>>>> (1) upload a catalog YAML file to /v1/catalog
> >>>>>>>>>
> >>>>>>>>> (2) install a bundle with catalog.bom in the root
> >>>>>>>>>
> >>>>>>>>> The feature (2) is disabled by default, but I'd like to move
> towards
> >>>>>>>>> enabling it.  This will make it easier to create nicely
> structured
> >>> BOM
> >>>>>>>>> files because scripts etc can be taken out of the BOM, stored as
> >>>> files in
> >>>>>>>>> the same bundle.  (Because URLs of the form
> >>>>>>>>> `classpath://scripts/install.sh`  use the bundle's classpath to
> >>>> resolve.)
> >>>>>>>>> As a first step in #485 [1] I do a few things:
> >>>>>>>>>
> >>>>>>>>> (A) add a utility class BundleMaker that lets us create and
> modify
> >>>>>>>>> bundles/zips, to make it easier to do things we might want to
> with
> >>>>>>>> bundles,
> >>>>>>>>> especially for testing
> >>>>>>>>>
> >>>>>>>>> (B) add an endpoint to the REST API which allows uploading a
> bundle
> >>>> ZIP
> >>>>>>>>> (C) accept bundle symbolic name and version in that REST API to
> >>>>>>>> facilitate
> >>>>>>>>> uploading non-bundle ZIPs where the OSGi MANIFEST.MF is
> >>> automatically
> >>>>>>>>> generated
> >>>>>>>>>
> >>>>>>>>> With this PR, if you have a directory on your local file system
> with
> >>>>>>>>> scripts and config files, and a BOM which refers to them, you can
> >>> just
> >>>>>>>> ZIP
> >>>>>>>>> that up an upload it, specifying the bundle name so that a YAML
> >>>> blueprint
> >>>>>>>>> author never needs to touch any java-isms.
> >>>>>>>>>
> >>>>>>>>> Where I see this going is a development workflow where a user can
> >>> edit
> >>>>>>>>> files locally and upload the ZIP to have that installed, and if
> they
> >>>> make
> >>>>>>>>> changes locally they can POST it again to have catalog items
> updated
> >>>>>>>>> (because default version is a SNAPSHOT).  We could also:
> >>>>>>>>>
> >>>>>>>>> (D) have `br catalog add ~/my/project/ --name my.project` create
> a
> >>> ZIP
> >>>>>>>> and
> >>>>>>>>> POST it, with bundle name metadata, so essentially the user's
> >>> process
> >>>> is
> >>>>>>>>> just to run that whenever they make a change
> >>>>>>>>>
> >>>>>>>>> (E) have a mechanism whereby deployed entities based on an
> affected
> >>>>>>>>> blueprint are optionally migrated to the new code, so if you've
> >>>> changed
> >>>>>>>> an
> >>>>>>>>> enricher the changes are picked up, or if say a launch.sh script
> has
> >>>>>>>>> changed, a restart will run the new code
> >>>>>>>>>
> >>>>>>>>> The above are fairly straightforward programmatically (although
> good
> >>>> user
> >>>>>>>>> interaction with (E) needs some thought).  So I think we can
> pretty
> >>>>>>>> quickly
> >>>>>>>>> get to a much smoother dev workflow.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> That's the highlight of this message.  You can jump to the end,
> >>> unless
> >>>>>>>>> you're interested in some important but low level details...
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> I'm also tempted by:
> >>>>>>>>>
> >>>>>>>>> (F) Integration with web-based IDE and/or Brooklyn reading and
> >>> writing
> >>>>>>>>> straight from GitHub -- but this seems like a lot of work and I'm
> >>> not
> >>>>>>>>> convinced it's much better than (D) workflow-wise
> >>>>>>>>>
> >>>>>>>>> Before we can change (2) to be the default, or start widely using
> >>> the
> >>>>>>>> POST
> >>>>>>>>> a ZIP feature, we need to sort out some issues to do with
> >>> persistence
> >>>> and
> >>>>>>>>> reloading:
> >>>>>>>>>
> >>>>>>>>> (G) Bundles installed via this mechanism are not persisted
> >>> currently,
> >>>> so
> >>>>>>>> if
> >>>>>>>>> you move to a different Brooklyn using the same backing store,
> >>> you'll
> >>>>>>>> lose
> >>>>>>>>> those bundles
> >>>>>>>>>
> >>>>>>>>> (H) On rebind, bundles aren't always activated when needed,
> meaning
> >>>> items
> >>>>>>>>> can't be loaded
> >>>>>>>>>
> >>>>>>>>> (I) We persist the individual catalog items as YAML, so we end up
> >>> with
> >>>>>>>> two
> >>>>>>>>> records — the YAML from the catalog.bom in the bundle, and the
> YAML
> >>>>>>>>> persisted for the item.  This isn't a problem per se, but
> something
> >>> to
> >>>>>>>>> think about, and some sometimes surprising behaviour.  In
> particular
> >>>> if
> >>>>>>>> you
> >>>>>>>>> delete the persisted YAML, the bundle is still there so the item
> is
> >>> no
> >>>>>>>>> longer deleted after a full rebind.
> >>>>>>>>>
> >>>>>>>>> One idea which might be useful is:
> >>>>>>>>>
> >>>>>>>>> (J) Introduce a catalogGroupId field on catalog items; this will
> do
> >>>> two
> >>>>>>>>> things:  if you try to delete an item with such a record, you'll
> be
> >>>>>>>>> encouraged to delete all such items (maybe disallowed to delete
> an
> >>>>>>>>> individual one), with the effect of deleting the bundle if it
> comes
> >>>> from
> >>>>>>>> a
> >>>>>>>>> bundle; and when resolving types we search first for items with
> the
> >>>> same
> >>>>>>>>> catalogGroupId (so that e.g. if I install MyCluster:1.0 and
> >>>> MyNode:1.0 in
> >>>>>>>>> the same group, the former can refer simply to "MyNode" but if I
> >>>> install
> >>>>>>>> a
> >>>>>>>>> 2.0 version of that group, the 1.0 cluster still loads the 1.0
> node
> >>> --
> >>>>>>>> this
> >>>>>>>>> has bitten people i the past)
> >>>>>>>>>
> >>>>>>>>> There is a related Brooklyn upgrade problem worth mentioning,
> which
> >>>> the
> >>>>>>>>> above might help with, where:
> >>>>>>>>>
> >>>>>>>>> (K) If I migrate from Brooklyn 10 to 11 when it comes out, I'll
> no
> >>>> longer
> >>>>>>>>> have certain entities that were at v10, since we don't include
> >>> those;
> >>>> an
> >>>>>>>>> upgrade could include rules that certain groupIds need to be
> >>> updated,
> >>>> or
> >>>>>>>> it
> >>>>>>>>> can search and attempt to automatically apply the updates
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Quite a lot here and we don't need to solve it but I wanted to:
> >>>>>>>>>
> >>>>>>>>> * Share the current thinking
> >>>>>>>>>
> >>>>>>>>> * Get opinions on the general dev workflow suggested by (D)
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Thanks for feedback -- and if we like it help with (D) would be
> >>>>>>>> appreciated!
> >>>>>>>>> Best
> >>>>>>>>> Alex
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> [1] . https://github.com/apache/brooklyn-server/pull/485
> >>>
> >
>
> --

Thomas Bouron • Software Engineer @ Cloudsoft Corporation •
http://www.cloudsoftcorp.com/
Github: https://github.com/tbouron
Twitter: https://twitter.com/eltibouron

Re: Uploading ZIPs for a better dev workflow

[Svetoslav Neykov <sv...@cloudsoftcorp.com>]

Having quickly scanned the thread I still don't see a reason why the requirement to have a consistent naming for the bundle itself which is not user visible? Could instead have behaviour similar to the machinNamer here.

Slight preference for optional symbolicName when it's a jar file. Though I really like Geoff's idea of "one true way". The workflow to create a jar file is very different from creating a zip file (either from a folder or user supplied) so don't think they conflict.

Svet.


> On 7.03.2017 г., at 17:47, Alex Heneveld <al...@cloudsoftcorp.com> wrote:
> 
> 
> Considering the points made I'm pretty sure it's the case that requiring a symbolic name and version in the BOM presents little if any unnecessary burden, in which case I'm persuaded of the "one true way" philosophy.  I really like it.  :)
> 
> Can anyone think of a case where it would be a burden?  My thinking is that:
> 
> * If you're uploading a ZIP of YAML and scripts and icons, you have a catalog.bom and that's a useful place to require the symname and version
> * If your ZIP/JAR also has a MANIFEST.MF the symname and version will be present in it also; the user is duplicating that info (this is the "little" unnecssary burden) but it's tiny and the benefit of one true way dominates; we do require that they match
> * If your ZIP/JAR doesn't have a BOM there's no point in uploading it to the catalog endpoint; either it's a JAR but this isn't meant as an "install arbitrary bundles" endpoint, or a non-JAR in which case what are we supposed to do with it once you upload it?
> 
> It keeps the contract simple, and the CLI very simple.   `br push /path/to/blueprint/x/`
> (But Geoff as a side point I think the CLI should _not_ attempt validation of the input/version -- the server will do that anyway and no benefit in doing it twice, but more code to maintain.)
> 
> I'm further persuaded given it is in a file, we should require a version.  I was concerned about needing to supply the version every time on the CLI but this has gone away.
> 
> I see no reason for a separate "manifest" section.  Removing this eliminates the prospect of Aled's pathological/"perverted" example.  I favour:
> 
>    brooklyn.catalog:
>       symbolicName: com.example.Foo
>       version: 1.0
>       items:
>       - id: foo
>         version: 3.0   # optional version override
>         itemType: entity
>         item:
>           type: blah
>           ...
> 
> That is, the fields symbolic-name and version are _required_ under the `brooklyn.catalog` when POSTing a ZIP to the catalog endpoint, and are used to define the bundle.  The version (and icon, etc) are then inherited by default by items defined therein.  If the ZIP is a JAR which contains a MANIFEST.MF, the bundle information (symbolic name and version) must match exactly.
> 
> Sound right?
> 
> Best
> Alex
> 
> 
> On 03/03/2017 09:59, Geoff Macartney wrote:
>> It also means clients ('br' at least) can easily validate the upload even
>> before execution by checking that all the manifest details are present.
>> 
>> On Fri, 3 Mar 2017 at 09:57 Geoff Macartney <
>> geoff.macartney@cloudsoftcorp.com> wrote:
>> 
>>> Just on the last point of "We could also support the example below", I'd
>>> say let's not even be that flexible - my feeling is that the more flexible
>>> we are, the more confusing it is, and the harder to get right.  If we're
>>> going to add this capability let's just have one way to do it; if you are
>>> going to do a ZIP upload, you MUST have a catalog.bom, which MUST contain a
>>> manifest section, which MUST have both a symbolic name and a version.  That
>>> way everything's explicit and very clear, every time.
>>> 
>>> 
>>> 
>>> On Thu, 2 Mar 2017 at 11:45 Aled Sage <al...@gmail.com> wrote:
>>> 
>>> Geoff, all,
>>> 
>>> I was imagining the manifest version (in the catalog.bom) to be separate
>>> from the item version. The reason is that we support multiple items in
>>> the bom that can be independently versioned.
>>> 
>>> Somone perverted could write:
>>> 
>>>     brooklyn.catalog:
>>>        version: 1.0
>>>        manifest:
>>>          symbolic_name: com.example.Foo
>>>          version: 2.0
>>>        items:
>>>        - id: foo
>>>          version: 3.0
>>>          itemType: entity
>>>          item:
>>>            type: blah
>>>        - id: bar
>>>          version: 4.0
>>>          itemType: entity
>>>          item:
>>>         type: blah.blah
>>> 
>>> Here, the "1.0" version is not used by anything; the auto-generated OSGi
>>> bundle would be version 2.0; catalog item foo would be 3.0; and catalog
>>> item bar would be 4.0.
>>> 
>>> If we were starting from scratch, I'd be tempted to lock things down a
>>> lot more for what we actually support. For now, it feels like asking for
>>> an explicit version is a reasonable thing to do.
>>> 
>>> ---
>>> We could also support the example below, where "1.0" is used by both the
>>> manifest and the item:
>>> 
>>>     brooklyn.catalog:
>>>        version: 1.0
>>>        manifest:
>>>          symbolic_name: com.example.Foo
>>>        items:
>>>        - id: foo
>>>          itemType: entity
>>>          item:
>>>            type: blah
>>> 
>>> The general rule would be: if there is a version in the manifest
>>> section, that will be used; if not then a top-level version number will
>>> be used. If that is also missing, then fail.
>>> 
>>> i.e. we would not accept:
>>> 
>>>     # Fails - manifest has no version.
>>>     brooklyn.catalog:
>>>        manifest:
>>>          symbolic_name: com.example.Foo
>>>        items:
>>>        - id: foo
>>>          version: 1.0
>>>          itemType: entity
>>>          item:
>>>            type: blah
>>> 
>>> Thoughts?
>>> 
>>> Aled
>>> 
>>> 
>>> On 02/03/2017 10:02, Geoff Macartney wrote:
>>>> I take Alex's point about the manifest being Java specific, and I agree
>>>> therefore we shouldn't insist on it.
>>>> 
>>>> +1 also to preferring explicit name/version in the catalog.bom rather
>>> than
>>>> as API params, I agree we do
>>>> need to keep the version in source control.
>>>> 
>>>> Question on your straw man, does the 'version' below
>>>> 
>>>>      brooklyn.catalog:
>>>>         manifest:
>>>>           symbolic_name: com.example.Foo
>>>>           version: 1.0.0-SNAPSHOT
>>>>         items:
>>>> 
>>>> _replace_ the 'version' in existing catalogs
>>>> 
>>>> brooklyn.catalog:
>>>>      version: "0.1.0-SNAPSHOT"
>>>> 
>>>> or is it an independently variable value, i.e. the version of the bundle,
>>>> separate from the version of catalog items that it happens to contain?
>>>> (Sounds a bit disquieting, but I guess it's possible).
>>>> 
>>>> e.g. could you have
>>>> 
>>>> brooklyn.catalog:
>>>>    version: 0.11.0-SNAPSHOT
>>>>    manifest:
>>>>      symbolic_name: com.example.Foo
>>>>      version: 1.0.0-SNAPSHOT
>>>>    items:
>>>>    - id: foo
>>>>      type: xyz
>>>> 
>>>> which would be bundle com.example.Foo:1.0.0-SNAPSHOT, which happens to
>>>> contain item foo:0.11.0-SNAPSHOT?
>>>> 
>>>> If so, what would happen if you left out the second line above? What
>>>> version of 'foo' would the catalog contain?
>>>> 
>>>> Or does the version within the manifest _mean_ that it is not only the
>>>> bundle version you are specifying, but the catalog item versions too? (I
>>>> guess unless the item explicitly supplies its own version.)
>>>> 
>>>> Geoff
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Wed, 1 Mar 2017 at 17:26 Aled Sage <al...@gmail.com> wrote:
>>>> 
>>>> Hi all,
>>>> 
>>>> Discussion of the REST api kicked off again in
>>>> 
>>> https://github.com/apache/brooklyn-server/pull/485#issuecomment-283280366:
>>>>      Alex wrote:
>>>> 
>>>>      Requiring a MANIFEST.MF makes the input strongly java-centric; I'd
>>>>      like to appeal to people who write YAML blueprints with co-bundled
>>>>      scripts and images. They'd wonder why they can't simply make and
>>>>      upload a ZIP. They could probably be persuaded to supply a name and
>>>>      optional version on a CLI or UI, and understand why it is needed
>>>>      (hence supporting those args) but it would not be idiomatic to
>>>>      anyone but a java programmer to create a META-INF/ dir with a
>>>>      MANIFEST.MF and its syntax. Meanwhile it is very easy for us to
>>>>      convert the ZIP to a JAR on the server. Feels like uncontroversial
>>>>      good UX.
>>>> 
>>>>      OTOH for a java programmer a MANIFEST.MF is natural, and they'd want
>>>>      to drop the name/version args if they are in that file, and I see no
>>>>      reason to forbid that pattern.
>>>> 
>>>> I agree with Alex, that we should not require a META-INF/MANIFEST.MF. As
>>>> for Geoff's suggestion that we could auto-generate a manifest in the
>>>> `br` CLI, I'd prefer a more general solution that works for users of the
>>>> REST api as well (i.e. doing it server-side).
>>>> 
>>>> ---
>>>> Svet suggested that the catalog.bom could give the symbolic name +
>>>> version, via additional metadata in that file.
>>>> 
>>>> What I really like about that is it's in version control, becuase it's
>>>> in the file/repo. If alternatively the name/version are just passed as
>>>> REST api parameters, then it's not in version control (and is more
>>>> susceptible to typos etc).
>>>> 
>>>> I'm not sure what we'd want this to look like. As a straw man:
>>>> 
>>>>      brooklyn.catalog:
>>>>         manifest:
>>>>           symbolic_name: com.example.Foo
>>>>           version: 1.0.0-SNAPSHOT
>>>>         items:
>>>>         - ...
>>>> 
>>>> If the user built their own real OSGi bundle, then they wouldn't need to
>>>> include this "manifest" section. If they did include that and it
>>>> contradicted the OSGi bundle's manifest, then we'd fail-fast.
>>>> 
>>>> (Note this reminds me of the (unrelated) metadata described in
>>>> https://github.com/brooklyncentral/blueprint-repository- there is a
>>>> "publish" block that can be added to the catalog.bom.)
>>>> 
>>>> ---
>>>> With Svet's suggestion, if there was no manifest section/file, then
>>>> should we auto-generate something from the item(s) in the catalog.bom?
>>>> 
>>>> I can see how that could easily work for a .bom file that has a single
>>>> item (e.g. we use the catalog item's id + version, possibly with a
>>>> special prefix in the symbolic name to avoid conflicts).
>>>> 
>>>> However, if there are multiple items then it would get trickier.
>>>> 
>>>> I'm inclined to say that for a minimal viable product (MVP) we can
>>>> insist on the "manifest" section in the catalog.bom.
>>>> 
>>>> Aled
>>>> 
>>>> 
>>>> On 20/12/2016 16:34, Svetoslav Neykov wrote:
>>>>>> Svet, if instead we tried to infer it from the catalog.bom, would we
>>>> require some additional metadata within the .bom file? Or would we use
>>> the
>>>> catalog item's id + version? I'm not convinced by the latter - it would
>>>> mean some .bom files would work and others wouldn't (e.g. if the .bom had
>>>> multiple items with different versions). Better to support the explicit
>>>> approach IMO.
>>>>> I imagine it would be additional metadata. On the other hand I don't see
>>>> a technical reason why we need an explicit symbolicName and version -
>>> they
>>>> can be auto-generated.
>>>>> Svet.
>>>>> 
>>>>> 
>>>>>> On 20.12.2016 г., at 17:50, Aled Sage <al...@gmail.com> wrote:
>>>>>> 
>>>>>> Hi all,
>>>>>> 
>>>>>> +1
>>>>>> 
>>>>>> (D) sounds good. What version are you imagining the bundle would be, if
>>>> one runs `br catalog add ~/my/project/ --name com.example.myproject`?
>>>>>> ---
>>>>>> I like the idea of uploading a plain zip (rather than only supporting
>>>> OSGi bundles) - that makes it simpler for non-java folk. The use of OSGi
>>>> becomes a (hidden) implementation detail to many users.
>>>>>> ---
>>>>>> If auto-generating the manifest, I think we need the user to be
>>> explicit
>>>> about symbolic name and version. Having these supplied in the REST api
>>> call
>>>> (as Alex suggests) would achieve that.
>>>>>> Svet, if instead we tried to infer it from the catalog.bom, would we
>>>> require some additional metadata within the .bom file? Or would we use
>>> the
>>>> catalog item's id + version? I'm not convinced by the latter - it would
>>>> mean some .bom files would work and others wouldn't (e.g. if the .bom had
>>>> multiple items with different versions). Better to support the explicit
>>>> approach IMO.
>>>>>> ---
>>>>>> For E ("have a mechanism whereby deployed entities based on an affected
>>>> blueprint are optionally migrated to the new code"), that feels like a
>>>> separate discussion. It could equally apply to a pure YAML .bom file that
>>>> has been added to the catalog.
>>>>>> I suggest we discuss that in a separate email thread.
>>>>>> 
>>>>>> ---
>>>>>> For (G), it's an interesting suggestion from Svet to make use of Karaf
>>>> Cellar for HA nodes. I'm hesitant (e.g. if restarting a standalone
>>> Brooklyn
>>>> node whose VM has died, then it adds big additional requirements for what
>>>> constitutes the "persisted state"). On the other hand, it's good to use
>>>> well-established technologies rather than re-inventing things!
>>>>>> An alternative ("pure brooklyn") approach could be to write the bundle
>>>> to persisted state; on rebind, we'd install + activate those bundles.
>>>>>> ---
>>>>>> For "catalogGroupId", I agree with Svet that in the initial use-case
>>>> this can be an implementation detail.
>>>>>> It could be set as the bundle's symbolic name + version: everything
>>> from
>>>> the bundle should be deleted at once, along with the bundle.
>>>>>> Longer term, I can see how exposing "catalogGroupId" to the user could
>>>> support more use-cases (e.g. for several catalog items from different
>>>> bundles to work together). I don't think we should try to support that
>>> yet.
>>>>>> Aled
>>>>>> 
>>>>>> 
>>>>>> On 19/12/2016 17:19, Geoff Macartney wrote:
>>>>>>> hi Alex,
>>>>>>> 
>>>>>>> this looks like a good feature to have, I shall look at the PR as soon
>>>> as I
>>>>>>> can.
>>>>>>> 
>>>>>>> The catalog.bom scanner feature was initially enabled by default, but
>>> we
>>>>>>> had to
>>>>>>> disable it because it turned out not to work properly with rebind. I
>>>> don't
>>>>>>> think
>>>>>>> it should be a lot of work to fix that but it hasn't been something
>>>> we've
>>>>>>> got round
>>>>>>> to yet.  This would be a great opportunity to look back at that.
>>>>>>> 
>>>>>>> Some random thoughts:
>>>>>>> 
>>>>>>> re (C), if we are going to treat the zips as bundles, my gut feel is
>>>> that
>>>>>>> we
>>>>>>> should insist on a manifest and get the metadata from it.  It doesn't
>>>> feel
>>>>>>> to me
>>>>>>> like it makes much sense to allow a zip file without a MANIFEST.MF but
>>>>>>> convey
>>>>>>> the intended bundle metadata to Brooklyn via HTTP headers.  And rather
>>>> than
>>>>>>> infer bundle metadata I think it's better to ask users to be explicit
>>>> about
>>>>>>> what
>>>>>>> their intentions are.  To make users lives easier, we could
>>>>>>> add a command to br to generate the manifest (locally) with correct
>>>> syntax,
>>>>>>> so that the manifest is in the right place, rather than have br add
>>> the
>>>> data
>>>>>>> to the "upload" request headers.
>>>>>>> 
>>>>>>> re. (D) will be glad to have a look at it
>>>>>>> 
>>>>>>> re. (E) it would certainly need to be optional - maybe keep it as an
>>>>>>> explicit
>>>>>>> separate command ('upgrade'?)
>>>>>>> 
>>>>>>> (F) it does seem like a lot of work but might be nice for users who
>>> are
>>>> not
>>>>>>> keen on command lines.
>>>>>>> 
>>>>>>> G - I:  we'll definitely need to pay close attention to persistence
>>> and
>>>>>>> rebind;
>>>>>>> I wonder also about HA operation, are there any additional
>>> implications?
>>>>>>> (J) I think it would be good to treat all the files from a jar, sorry
>>>>>>> bundle,
>>>>>>> as an atomic group - cleaner that way perhaps than allowing
>>>> delete/update
>>>>>>> of
>>>>>>> individual entries from a bundle on a piecemeal basis.  Rest support
>>> on
>>>>>>> delete
>>>>>>> catalog could warn about related catalog entries being deleted and ask
>>>> for
>>>>>>> a "--force" param to confirm.
>>>>>>> 
>>>>>>> Geoff
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Fri, 16 Dec 2016 at 15:24 Svetoslav Neykov <
>>>>>>> svetoslav.neykov@cloudsoftcorp.com> wrote:
>>>>>>> 
>>>>>>>> +1
>>>>>>>> 
>>>>>>>> Some thoughts:
>>>>>>>>     * (A) add a utility class BundleMaker
>>>>>>>>       Sounds very similar to
>>>>>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles <
>>>>>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles>
>>>>>>>>       Looking at the code it's much more focused on zip files so I
>>> guess
>>>>>>>> there's no much overlap, but worth keeping in mind
>>>>>>>>     * (C) accept bundle symbolic name and version
>>>>>>>>       Why require them at all? Could infer them from the catalog.bom
>>> in
>>>> some
>>>>>>>> way - maybe require those properties to be in there. If not present
>>> are
>>>>>>>> they really needed?
>>>>>>>>     * (G) Bundles installed via this mechanism are not persisted
>>>> currently &
>>>>>>>> (I) We persist the individual catalog items as YAML, so we end up
>>> with
>>>> two
>>>>>>>> records
>>>>>>>>       Suggest marking the catalog items coming from bundles as
>>>>>>>> non-persistable. Then try to share the bundles between HA nodes.
>>> (Karaf
>>>>>>>> Cellar?)
>>>>>>>>     * (J) Introduce a catalogGroupId field on catalog items;
>>>>>>>>       Agree this could be useful and I like the idea of deleting the
>>>> bundle
>>>>>>>> altogether with the catalog items. From user's perspective I don't
>>> see
>>>> the
>>>>>>>> need for an extra field (i.e. it's an implementation detail).
>>>>>>>> 
>>>>>>>> Svet.
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On 16.12.2016 г., at 12:50, Alex Heneveld <
>>>>>>>> alex.heneveld@cloudsoftcorp.com> wrote:
>>>>>>>>> Hi Brooklyners-
>>>>>>>>> 
>>>>>>>>> In the code we currently have two routes for users to install new
>>>>>>>>> blueprints:
>>>>>>>>> 
>>>>>>>>> (1) upload a catalog YAML file to /v1/catalog
>>>>>>>>> 
>>>>>>>>> (2) install a bundle with catalog.bom in the root
>>>>>>>>> 
>>>>>>>>> The feature (2) is disabled by default, but I'd like to move towards
>>>>>>>>> enabling it.  This will make it easier to create nicely structured
>>> BOM
>>>>>>>>> files because scripts etc can be taken out of the BOM, stored as
>>>> files in
>>>>>>>>> the same bundle.  (Because URLs of the form
>>>>>>>>> `classpath://scripts/install.sh`  use the bundle's classpath to
>>>> resolve.)
>>>>>>>>> As a first step in #485 [1] I do a few things:
>>>>>>>>> 
>>>>>>>>> (A) add a utility class BundleMaker that lets us create and modify
>>>>>>>>> bundles/zips, to make it easier to do things we might want to with
>>>>>>>> bundles,
>>>>>>>>> especially for testing
>>>>>>>>> 
>>>>>>>>> (B) add an endpoint to the REST API which allows uploading a bundle
>>>> ZIP
>>>>>>>>> (C) accept bundle symbolic name and version in that REST API to
>>>>>>>> facilitate
>>>>>>>>> uploading non-bundle ZIPs where the OSGi MANIFEST.MF is
>>> automatically
>>>>>>>>> generated
>>>>>>>>> 
>>>>>>>>> With this PR, if you have a directory on your local file system with
>>>>>>>>> scripts and config files, and a BOM which refers to them, you can
>>> just
>>>>>>>> ZIP
>>>>>>>>> that up an upload it, specifying the bundle name so that a YAML
>>>> blueprint
>>>>>>>>> author never needs to touch any java-isms.
>>>>>>>>> 
>>>>>>>>> Where I see this going is a development workflow where a user can
>>> edit
>>>>>>>>> files locally and upload the ZIP to have that installed, and if they
>>>> make
>>>>>>>>> changes locally they can POST it again to have catalog items updated
>>>>>>>>> (because default version is a SNAPSHOT).  We could also:
>>>>>>>>> 
>>>>>>>>> (D) have `br catalog add ~/my/project/ --name my.project` create a
>>> ZIP
>>>>>>>> and
>>>>>>>>> POST it, with bundle name metadata, so essentially the user's
>>> process
>>>> is
>>>>>>>>> just to run that whenever they make a change
>>>>>>>>> 
>>>>>>>>> (E) have a mechanism whereby deployed entities based on an affected
>>>>>>>>> blueprint are optionally migrated to the new code, so if you've
>>>> changed
>>>>>>>> an
>>>>>>>>> enricher the changes are picked up, or if say a launch.sh script has
>>>>>>>>> changed, a restart will run the new code
>>>>>>>>> 
>>>>>>>>> The above are fairly straightforward programmatically (although good
>>>> user
>>>>>>>>> interaction with (E) needs some thought).  So I think we can pretty
>>>>>>>> quickly
>>>>>>>>> get to a much smoother dev workflow.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> That's the highlight of this message.  You can jump to the end,
>>> unless
>>>>>>>>> you're interested in some important but low level details...
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> I'm also tempted by:
>>>>>>>>> 
>>>>>>>>> (F) Integration with web-based IDE and/or Brooklyn reading and
>>> writing
>>>>>>>>> straight from GitHub -- but this seems like a lot of work and I'm
>>> not
>>>>>>>>> convinced it's much better than (D) workflow-wise
>>>>>>>>> 
>>>>>>>>> Before we can change (2) to be the default, or start widely using
>>> the
>>>>>>>> POST
>>>>>>>>> a ZIP feature, we need to sort out some issues to do with
>>> persistence
>>>> and
>>>>>>>>> reloading:
>>>>>>>>> 
>>>>>>>>> (G) Bundles installed via this mechanism are not persisted
>>> currently,
>>>> so
>>>>>>>> if
>>>>>>>>> you move to a different Brooklyn using the same backing store,
>>> you'll
>>>>>>>> lose
>>>>>>>>> those bundles
>>>>>>>>> 
>>>>>>>>> (H) On rebind, bundles aren't always activated when needed, meaning
>>>> items
>>>>>>>>> can't be loaded
>>>>>>>>> 
>>>>>>>>> (I) We persist the individual catalog items as YAML, so we end up
>>> with
>>>>>>>> two
>>>>>>>>> records — the YAML from the catalog.bom in the bundle, and the YAML
>>>>>>>>> persisted for the item.  This isn't a problem per se, but something
>>> to
>>>>>>>>> think about, and some sometimes surprising behaviour.  In particular
>>>> if
>>>>>>>> you
>>>>>>>>> delete the persisted YAML, the bundle is still there so the item is
>>> no
>>>>>>>>> longer deleted after a full rebind.
>>>>>>>>> 
>>>>>>>>> One idea which might be useful is:
>>>>>>>>> 
>>>>>>>>> (J) Introduce a catalogGroupId field on catalog items; this will do
>>>> two
>>>>>>>>> things:  if you try to delete an item with such a record, you'll be
>>>>>>>>> encouraged to delete all such items (maybe disallowed to delete an
>>>>>>>>> individual one), with the effect of deleting the bundle if it comes
>>>> from
>>>>>>>> a
>>>>>>>>> bundle; and when resolving types we search first for items with the
>>>> same
>>>>>>>>> catalogGroupId (so that e.g. if I install MyCluster:1.0 and
>>>> MyNode:1.0 in
>>>>>>>>> the same group, the former can refer simply to "MyNode" but if I
>>>> install
>>>>>>>> a
>>>>>>>>> 2.0 version of that group, the 1.0 cluster still loads the 1.0 node
>>> --
>>>>>>>> this
>>>>>>>>> has bitten people i the past)
>>>>>>>>> 
>>>>>>>>> There is a related Brooklyn upgrade problem worth mentioning, which
>>>> the
>>>>>>>>> above might help with, where:
>>>>>>>>> 
>>>>>>>>> (K) If I migrate from Brooklyn 10 to 11 when it comes out, I'll no
>>>> longer
>>>>>>>>> have certain entities that were at v10, since we don't include
>>> those;
>>>> an
>>>>>>>>> upgrade could include rules that certain groupIds need to be
>>> updated,
>>>> or
>>>>>>>> it
>>>>>>>>> can search and attempt to automatically apply the updates
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Quite a lot here and we don't need to solve it but I wanted to:
>>>>>>>>> 
>>>>>>>>> * Share the current thinking
>>>>>>>>> 
>>>>>>>>> * Get opinions on the general dev workflow suggested by (D)
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Thanks for feedback -- and if we like it help with (D) would be
>>>>>>>> appreciated!
>>>>>>>>> Best
>>>>>>>>> Alex
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> [1] . https://github.com/apache/brooklyn-server/pull/485
>>> 
> 

Re: Uploading ZIPs for a better dev workflow

[Alex Heneveld <al...@cloudsoftcorp.com>]

Considering the points made I'm pretty sure it's the case that requiring 
a symbolic name and version in the BOM presents little if any 
unnecessary burden, in which case I'm persuaded of the "one true way" 
philosophy.  I really like it.  :)

Can anyone think of a case where it would be a burden?  My thinking is that:

* If you're uploading a ZIP of YAML and scripts and icons, you have a 
catalog.bom and that's a useful place to require the symname and version
* If your ZIP/JAR also has a MANIFEST.MF the symname and version will be 
present in it also; the user is duplicating that info (this is the 
"little" unnecssary burden) but it's tiny and the benefit of one true 
way dominates; we do require that they match
* If your ZIP/JAR doesn't have a BOM there's no point in uploading it to 
the catalog endpoint; either it's a JAR but this isn't meant as an 
"install arbitrary bundles" endpoint, or a non-JAR in which case what 
are we supposed to do with it once you upload it?

It keeps the contract simple, and the CLI very simple.   `br push 
/path/to/blueprint/x/`
(But Geoff as a side point I think the CLI should _not_ attempt 
validation of the input/version -- the server will do that anyway and no 
benefit in doing it twice, but more code to maintain.)

I'm further persuaded given it is in a file, we should require a 
version.  I was concerned about needing to supply the version every time 
on the CLI but this has gone away.

I see no reason for a separate "manifest" section.  Removing this 
eliminates the prospect of Aled's pathological/"perverted" example.  I 
favour:

     brooklyn.catalog:
        symbolicName: com.example.Foo
        version: 1.0
        items:
        - id: foo
          version: 3.0   # optional version override
          itemType: entity
          item:
            type: blah
            ...

That is, the fields symbolic-name and version are _required_ under the 
`brooklyn.catalog` when POSTing a ZIP to the catalog endpoint, and are 
used to define the bundle.  The version (and icon, etc) are then 
inherited by default by items defined therein.  If the ZIP is a JAR 
which contains a MANIFEST.MF, the bundle information (symbolic name and 
version) must match exactly.

Sound right?

Best
Alex


On 03/03/2017 09:59, Geoff Macartney wrote:
> It also means clients ('br' at least) can easily validate the upload even
> before execution by checking that all the manifest details are present.
>
> On Fri, 3 Mar 2017 at 09:57 Geoff Macartney <
> geoff.macartney@cloudsoftcorp.com> wrote:
>
>> Just on the last point of "We could also support the example below", I'd
>> say let's not even be that flexible - my feeling is that the more flexible
>> we are, the more confusing it is, and the harder to get right.  If we're
>> going to add this capability let's just have one way to do it; if you are
>> going to do a ZIP upload, you MUST have a catalog.bom, which MUST contain a
>> manifest section, which MUST have both a symbolic name and a version.  That
>> way everything's explicit and very clear, every time.
>>
>>
>>
>> On Thu, 2 Mar 2017 at 11:45 Aled Sage <al...@gmail.com> wrote:
>>
>> Geoff, all,
>>
>> I was imagining the manifest version (in the catalog.bom) to be separate
>> from the item version. The reason is that we support multiple items in
>> the bom that can be independently versioned.
>>
>> Somone perverted could write:
>>
>>      brooklyn.catalog:
>>         version: 1.0
>>         manifest:
>>           symbolic_name: com.example.Foo
>>           version: 2.0
>>         items:
>>         - id: foo
>>           version: 3.0
>>           itemType: entity
>>           item:
>>             type: blah
>>         - id: bar
>>           version: 4.0
>>           itemType: entity
>>           item:
>>          type: blah.blah
>>
>> Here, the "1.0" version is not used by anything; the auto-generated OSGi
>> bundle would be version 2.0; catalog item foo would be 3.0; and catalog
>> item bar would be 4.0.
>>
>> If we were starting from scratch, I'd be tempted to lock things down a
>> lot more for what we actually support. For now, it feels like asking for
>> an explicit version is a reasonable thing to do.
>>
>> ---
>> We could also support the example below, where "1.0" is used by both the
>> manifest and the item:
>>
>>      brooklyn.catalog:
>>         version: 1.0
>>         manifest:
>>           symbolic_name: com.example.Foo
>>         items:
>>         - id: foo
>>           itemType: entity
>>           item:
>>             type: blah
>>
>> The general rule would be: if there is a version in the manifest
>> section, that will be used; if not then a top-level version number will
>> be used. If that is also missing, then fail.
>>
>> i.e. we would not accept:
>>
>>      # Fails - manifest has no version.
>>      brooklyn.catalog:
>>         manifest:
>>           symbolic_name: com.example.Foo
>>         items:
>>         - id: foo
>>           version: 1.0
>>           itemType: entity
>>           item:
>>             type: blah
>>
>> Thoughts?
>>
>> Aled
>>
>>
>> On 02/03/2017 10:02, Geoff Macartney wrote:
>>> I take Alex's point about the manifest being Java specific, and I agree
>>> therefore we shouldn't insist on it.
>>>
>>> +1 also to preferring explicit name/version in the catalog.bom rather
>> than
>>> as API params, I agree we do
>>> need to keep the version in source control.
>>>
>>> Question on your straw man, does the 'version' below
>>>
>>>       brooklyn.catalog:
>>>          manifest:
>>>            symbolic_name: com.example.Foo
>>>            version: 1.0.0-SNAPSHOT
>>>          items:
>>>
>>> _replace_ the 'version' in existing catalogs
>>>
>>> brooklyn.catalog:
>>>       version: "0.1.0-SNAPSHOT"
>>>
>>> or is it an independently variable value, i.e. the version of the bundle,
>>> separate from the version of catalog items that it happens to contain?
>>> (Sounds a bit disquieting, but I guess it's possible).
>>>
>>> e.g. could you have
>>>
>>> brooklyn.catalog:
>>>     version: 0.11.0-SNAPSHOT
>>>     manifest:
>>>       symbolic_name: com.example.Foo
>>>       version: 1.0.0-SNAPSHOT
>>>     items:
>>>     - id: foo
>>>       type: xyz
>>>
>>> which would be bundle com.example.Foo:1.0.0-SNAPSHOT, which happens to
>>> contain item foo:0.11.0-SNAPSHOT?
>>>
>>> If so, what would happen if you left out the second line above? What
>>> version of 'foo' would the catalog contain?
>>>
>>> Or does the version within the manifest _mean_ that it is not only the
>>> bundle version you are specifying, but the catalog item versions too? (I
>>> guess unless the item explicitly supplies its own version.)
>>>
>>> Geoff
>>>
>>>
>>>
>>>
>>> On Wed, 1 Mar 2017 at 17:26 Aled Sage <al...@gmail.com> wrote:
>>>
>>> Hi all,
>>>
>>> Discussion of the REST api kicked off again in
>>>
>> https://github.com/apache/brooklyn-server/pull/485#issuecomment-283280366:
>>>       Alex wrote:
>>>
>>>       Requiring a MANIFEST.MF makes the input strongly java-centric; I'd
>>>       like to appeal to people who write YAML blueprints with co-bundled
>>>       scripts and images. They'd wonder why they can't simply make and
>>>       upload a ZIP. They could probably be persuaded to supply a name and
>>>       optional version on a CLI or UI, and understand why it is needed
>>>       (hence supporting those args) but it would not be idiomatic to
>>>       anyone but a java programmer to create a META-INF/ dir with a
>>>       MANIFEST.MF and its syntax. Meanwhile it is very easy for us to
>>>       convert the ZIP to a JAR on the server. Feels like uncontroversial
>>>       good UX.
>>>
>>>       OTOH for a java programmer a MANIFEST.MF is natural, and they'd want
>>>       to drop the name/version args if they are in that file, and I see no
>>>       reason to forbid that pattern.
>>>
>>> I agree with Alex, that we should not require a META-INF/MANIFEST.MF. As
>>> for Geoff's suggestion that we could auto-generate a manifest in the
>>> `br` CLI, I'd prefer a more general solution that works for users of the
>>> REST api as well (i.e. doing it server-side).
>>>
>>> ---
>>> Svet suggested that the catalog.bom could give the symbolic name +
>>> version, via additional metadata in that file.
>>>
>>> What I really like about that is it's in version control, becuase it's
>>> in the file/repo. If alternatively the name/version are just passed as
>>> REST api parameters, then it's not in version control (and is more
>>> susceptible to typos etc).
>>>
>>> I'm not sure what we'd want this to look like. As a straw man:
>>>
>>>       brooklyn.catalog:
>>>          manifest:
>>>            symbolic_name: com.example.Foo
>>>            version: 1.0.0-SNAPSHOT
>>>          items:
>>>          - ...
>>>
>>> If the user built their own real OSGi bundle, then they wouldn't need to
>>> include this "manifest" section. If they did include that and it
>>> contradicted the OSGi bundle's manifest, then we'd fail-fast.
>>>
>>> (Note this reminds me of the (unrelated) metadata described in
>>> https://github.com/brooklyncentral/blueprint-repository- there is a
>>> "publish" block that can be added to the catalog.bom.)
>>>
>>> ---
>>> With Svet's suggestion, if there was no manifest section/file, then
>>> should we auto-generate something from the item(s) in the catalog.bom?
>>>
>>> I can see how that could easily work for a .bom file that has a single
>>> item (e.g. we use the catalog item's id + version, possibly with a
>>> special prefix in the symbolic name to avoid conflicts).
>>>
>>> However, if there are multiple items then it would get trickier.
>>>
>>> I'm inclined to say that for a minimal viable product (MVP) we can
>>> insist on the "manifest" section in the catalog.bom.
>>>
>>> Aled
>>>
>>>
>>> On 20/12/2016 16:34, Svetoslav Neykov wrote:
>>>>> Svet, if instead we tried to infer it from the catalog.bom, would we
>>> require some additional metadata within the .bom file? Or would we use
>> the
>>> catalog item's id + version? I'm not convinced by the latter - it would
>>> mean some .bom files would work and others wouldn't (e.g. if the .bom had
>>> multiple items with different versions). Better to support the explicit
>>> approach IMO.
>>>> I imagine it would be additional metadata. On the other hand I don't see
>>> a technical reason why we need an explicit symbolicName and version -
>> they
>>> can be auto-generated.
>>>> Svet.
>>>>
>>>>
>>>>> On 20.12.2016 \u0433., at 17:50, Aled Sage <al...@gmail.com> wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> +1
>>>>>
>>>>> (D) sounds good. What version are you imagining the bundle would be, if
>>> one runs `br catalog add ~/my/project/ --name com.example.myproject`?
>>>>> ---
>>>>> I like the idea of uploading a plain zip (rather than only supporting
>>> OSGi bundles) - that makes it simpler for non-java folk. The use of OSGi
>>> becomes a (hidden) implementation detail to many users.
>>>>> ---
>>>>> If auto-generating the manifest, I think we need the user to be
>> explicit
>>> about symbolic name and version. Having these supplied in the REST api
>> call
>>> (as Alex suggests) would achieve that.
>>>>> Svet, if instead we tried to infer it from the catalog.bom, would we
>>> require some additional metadata within the .bom file? Or would we use
>> the
>>> catalog item's id + version? I'm not convinced by the latter - it would
>>> mean some .bom files would work and others wouldn't (e.g. if the .bom had
>>> multiple items with different versions). Better to support the explicit
>>> approach IMO.
>>>>> ---
>>>>> For E ("have a mechanism whereby deployed entities based on an affected
>>> blueprint are optionally migrated to the new code"), that feels like a
>>> separate discussion. It could equally apply to a pure YAML .bom file that
>>> has been added to the catalog.
>>>>> I suggest we discuss that in a separate email thread.
>>>>>
>>>>> ---
>>>>> For (G), it's an interesting suggestion from Svet to make use of Karaf
>>> Cellar for HA nodes. I'm hesitant (e.g. if restarting a standalone
>> Brooklyn
>>> node whose VM has died, then it adds big additional requirements for what
>>> constitutes the "persisted state"). On the other hand, it's good to use
>>> well-established technologies rather than re-inventing things!
>>>>> An alternative ("pure brooklyn") approach could be to write the bundle
>>> to persisted state; on rebind, we'd install + activate those bundles.
>>>>> ---
>>>>> For "catalogGroupId", I agree with Svet that in the initial use-case
>>> this can be an implementation detail.
>>>>> It could be set as the bundle's symbolic name + version: everything
>> from
>>> the bundle should be deleted at once, along with the bundle.
>>>>> Longer term, I can see how exposing "catalogGroupId" to the user could
>>> support more use-cases (e.g. for several catalog items from different
>>> bundles to work together). I don't think we should try to support that
>> yet.
>>>>> Aled
>>>>>
>>>>>
>>>>> On 19/12/2016 17:19, Geoff Macartney wrote:
>>>>>> hi Alex,
>>>>>>
>>>>>> this looks like a good feature to have, I shall look at the PR as soon
>>> as I
>>>>>> can.
>>>>>>
>>>>>> The catalog.bom scanner feature was initially enabled by default, but
>> we
>>>>>> had to
>>>>>> disable it because it turned out not to work properly with rebind. I
>>> don't
>>>>>> think
>>>>>> it should be a lot of work to fix that but it hasn't been something
>>> we've
>>>>>> got round
>>>>>> to yet.  This would be a great opportunity to look back at that.
>>>>>>
>>>>>> Some random thoughts:
>>>>>>
>>>>>> re (C), if we are going to treat the zips as bundles, my gut feel is
>>> that
>>>>>> we
>>>>>> should insist on a manifest and get the metadata from it.  It doesn't
>>> feel
>>>>>> to me
>>>>>> like it makes much sense to allow a zip file without a MANIFEST.MF but
>>>>>> convey
>>>>>> the intended bundle metadata to Brooklyn via HTTP headers.  And rather
>>> than
>>>>>> infer bundle metadata I think it's better to ask users to be explicit
>>> about
>>>>>> what
>>>>>> their intentions are.  To make users lives easier, we could
>>>>>> add a command to br to generate the manifest (locally) with correct
>>> syntax,
>>>>>> so that the manifest is in the right place, rather than have br add
>> the
>>> data
>>>>>> to the "upload" request headers.
>>>>>>
>>>>>> re. (D) will be glad to have a look at it
>>>>>>
>>>>>> re. (E) it would certainly need to be optional - maybe keep it as an
>>>>>> explicit
>>>>>> separate command ('upgrade'?)
>>>>>>
>>>>>> (F) it does seem like a lot of work but might be nice for users who
>> are
>>> not
>>>>>> keen on command lines.
>>>>>>
>>>>>> G - I:  we'll definitely need to pay close attention to persistence
>> and
>>>>>> rebind;
>>>>>> I wonder also about HA operation, are there any additional
>> implications?
>>>>>> (J) I think it would be good to treat all the files from a jar, sorry
>>>>>> bundle,
>>>>>> as an atomic group - cleaner that way perhaps than allowing
>>> delete/update
>>>>>> of
>>>>>> individual entries from a bundle on a piecemeal basis.  Rest support
>> on
>>>>>> delete
>>>>>> catalog could warn about related catalog entries being deleted and ask
>>> for
>>>>>> a "--force" param to confirm.
>>>>>>
>>>>>> Geoff
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, 16 Dec 2016 at 15:24 Svetoslav Neykov <
>>>>>> svetoslav.neykov@cloudsoftcorp.com> wrote:
>>>>>>
>>>>>>> +1
>>>>>>>
>>>>>>> Some thoughts:
>>>>>>>      * (A) add a utility class BundleMaker
>>>>>>>        Sounds very similar to
>>>>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles <
>>>>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles>
>>>>>>>        Looking at the code it's much more focused on zip files so I
>> guess
>>>>>>> there's no much overlap, but worth keeping in mind
>>>>>>>      * (C) accept bundle symbolic name and version
>>>>>>>        Why require them at all? Could infer them from the catalog.bom
>> in
>>> some
>>>>>>> way - maybe require those properties to be in there. If not present
>> are
>>>>>>> they really needed?
>>>>>>>      * (G) Bundles installed via this mechanism are not persisted
>>> currently &
>>>>>>> (I) We persist the individual catalog items as YAML, so we end up
>> with
>>> two
>>>>>>> records
>>>>>>>        Suggest marking the catalog items coming from bundles as
>>>>>>> non-persistable. Then try to share the bundles between HA nodes.
>> (Karaf
>>>>>>> Cellar?)
>>>>>>>      * (J) Introduce a catalogGroupId field on catalog items;
>>>>>>>        Agree this could be useful and I like the idea of deleting the
>>> bundle
>>>>>>> altogether with the catalog items. From user's perspective I don't
>> see
>>> the
>>>>>>> need for an extra field (i.e. it's an implementation detail).
>>>>>>>
>>>>>>> Svet.
>>>>>>>
>>>>>>>
>>>>>>>> On 16.12.2016 \u0433., at 12:50, Alex Heneveld <
>>>>>>> alex.heneveld@cloudsoftcorp.com> wrote:
>>>>>>>> Hi Brooklyners-
>>>>>>>>
>>>>>>>> In the code we currently have two routes for users to install new
>>>>>>>> blueprints:
>>>>>>>>
>>>>>>>> (1) upload a catalog YAML file to /v1/catalog
>>>>>>>>
>>>>>>>> (2) install a bundle with catalog.bom in the root
>>>>>>>>
>>>>>>>> The feature (2) is disabled by default, but I'd like to move towards
>>>>>>>> enabling it.  This will make it easier to create nicely structured
>> BOM
>>>>>>>> files because scripts etc can be taken out of the BOM, stored as
>>> files in
>>>>>>>> the same bundle.  (Because URLs of the form
>>>>>>>> `classpath://scripts/install.sh`  use the bundle's classpath to
>>> resolve.)
>>>>>>>> As a first step in #485 [1] I do a few things:
>>>>>>>>
>>>>>>>> (A) add a utility class BundleMaker that lets us create and modify
>>>>>>>> bundles/zips, to make it easier to do things we might want to with
>>>>>>> bundles,
>>>>>>>> especially for testing
>>>>>>>>
>>>>>>>> (B) add an endpoint to the REST API which allows uploading a bundle
>>> ZIP
>>>>>>>> (C) accept bundle symbolic name and version in that REST API to
>>>>>>> facilitate
>>>>>>>> uploading non-bundle ZIPs where the OSGi MANIFEST.MF is
>> automatically
>>>>>>>> generated
>>>>>>>>
>>>>>>>> With this PR, if you have a directory on your local file system with
>>>>>>>> scripts and config files, and a BOM which refers to them, you can
>> just
>>>>>>> ZIP
>>>>>>>> that up an upload it, specifying the bundle name so that a YAML
>>> blueprint
>>>>>>>> author never needs to touch any java-isms.
>>>>>>>>
>>>>>>>> Where I see this going is a development workflow where a user can
>> edit
>>>>>>>> files locally and upload the ZIP to have that installed, and if they
>>> make
>>>>>>>> changes locally they can POST it again to have catalog items updated
>>>>>>>> (because default version is a SNAPSHOT).  We could also:
>>>>>>>>
>>>>>>>> (D) have `br catalog add ~/my/project/ --name my.project` create a
>> ZIP
>>>>>>> and
>>>>>>>> POST it, with bundle name metadata, so essentially the user's
>> process
>>> is
>>>>>>>> just to run that whenever they make a change
>>>>>>>>
>>>>>>>> (E) have a mechanism whereby deployed entities based on an affected
>>>>>>>> blueprint are optionally migrated to the new code, so if you've
>>> changed
>>>>>>> an
>>>>>>>> enricher the changes are picked up, or if say a launch.sh script has
>>>>>>>> changed, a restart will run the new code
>>>>>>>>
>>>>>>>> The above are fairly straightforward programmatically (although good
>>> user
>>>>>>>> interaction with (E) needs some thought).  So I think we can pretty
>>>>>>> quickly
>>>>>>>> get to a much smoother dev workflow.
>>>>>>>>
>>>>>>>>
>>>>>>>> That's the highlight of this message.  You can jump to the end,
>> unless
>>>>>>>> you're interested in some important but low level details...
>>>>>>>>
>>>>>>>>
>>>>>>>> I'm also tempted by:
>>>>>>>>
>>>>>>>> (F) Integration with web-based IDE and/or Brooklyn reading and
>> writing
>>>>>>>> straight from GitHub -- but this seems like a lot of work and I'm
>> not
>>>>>>>> convinced it's much better than (D) workflow-wise
>>>>>>>>
>>>>>>>> Before we can change (2) to be the default, or start widely using
>> the
>>>>>>> POST
>>>>>>>> a ZIP feature, we need to sort out some issues to do with
>> persistence
>>> and
>>>>>>>> reloading:
>>>>>>>>
>>>>>>>> (G) Bundles installed via this mechanism are not persisted
>> currently,
>>> so
>>>>>>> if
>>>>>>>> you move to a different Brooklyn using the same backing store,
>> you'll
>>>>>>> lose
>>>>>>>> those bundles
>>>>>>>>
>>>>>>>> (H) On rebind, bundles aren't always activated when needed, meaning
>>> items
>>>>>>>> can't be loaded
>>>>>>>>
>>>>>>>> (I) We persist the individual catalog items as YAML, so we end up
>> with
>>>>>>> two
>>>>>>>> records \u2014 the YAML from the catalog.bom in the bundle, and the YAML
>>>>>>>> persisted for the item.  This isn't a problem per se, but something
>> to
>>>>>>>> think about, and some sometimes surprising behaviour.  In particular
>>> if
>>>>>>> you
>>>>>>>> delete the persisted YAML, the bundle is still there so the item is
>> no
>>>>>>>> longer deleted after a full rebind.
>>>>>>>>
>>>>>>>> One idea which might be useful is:
>>>>>>>>
>>>>>>>> (J) Introduce a catalogGroupId field on catalog items; this will do
>>> two
>>>>>>>> things:  if you try to delete an item with such a record, you'll be
>>>>>>>> encouraged to delete all such items (maybe disallowed to delete an
>>>>>>>> individual one), with the effect of deleting the bundle if it comes
>>> from
>>>>>>> a
>>>>>>>> bundle; and when resolving types we search first for items with the
>>> same
>>>>>>>> catalogGroupId (so that e.g. if I install MyCluster:1.0 and
>>> MyNode:1.0 in
>>>>>>>> the same group, the former can refer simply to "MyNode" but if I
>>> install
>>>>>>> a
>>>>>>>> 2.0 version of that group, the 1.0 cluster still loads the 1.0 node
>> --
>>>>>>> this
>>>>>>>> has bitten people i the past)
>>>>>>>>
>>>>>>>> There is a related Brooklyn upgrade problem worth mentioning, which
>>> the
>>>>>>>> above might help with, where:
>>>>>>>>
>>>>>>>> (K) If I migrate from Brooklyn 10 to 11 when it comes out, I'll no
>>> longer
>>>>>>>> have certain entities that were at v10, since we don't include
>> those;
>>> an
>>>>>>>> upgrade could include rules that certain groupIds need to be
>> updated,
>>> or
>>>>>>> it
>>>>>>>> can search and attempt to automatically apply the updates
>>>>>>>>
>>>>>>>>
>>>>>>>> Quite a lot here and we don't need to solve it but I wanted to:
>>>>>>>>
>>>>>>>> * Share the current thinking
>>>>>>>>
>>>>>>>> * Get opinions on the general dev workflow suggested by (D)
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks for feedback -- and if we like it help with (D) would be
>>>>>>> appreciated!
>>>>>>>> Best
>>>>>>>> Alex
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> [1] . https://github.com/apache/brooklyn-server/pull/485
>>

Re: Uploading ZIPs for a better dev workflow

[Geoff Macartney <ge...@cloudsoftcorp.com>]

It also means clients ('br' at least) can easily validate the upload even
before execution by checking that all the manifest details are present.

On Fri, 3 Mar 2017 at 09:57 Geoff Macartney <
geoff.macartney@cloudsoftcorp.com> wrote:

> Just on the last point of "We could also support the example below", I'd
> say let's not even be that flexible - my feeling is that the more flexible
> we are, the more confusing it is, and the harder to get right.  If we're
> going to add this capability let's just have one way to do it; if you are
> going to do a ZIP upload, you MUST have a catalog.bom, which MUST contain a
> manifest section, which MUST have both a symbolic name and a version.  That
> way everything's explicit and very clear, every time.
>
>
>
> On Thu, 2 Mar 2017 at 11:45 Aled Sage <al...@gmail.com> wrote:
>
> Geoff, all,
>
> I was imagining the manifest version (in the catalog.bom) to be separate
> from the item version. The reason is that we support multiple items in
> the bom that can be independently versioned.
>
> Somone perverted could write:
>
>     brooklyn.catalog:
>        version: 1.0
>        manifest:
>          symbolic_name: com.example.Foo
>          version: 2.0
>        items:
>        - id: foo
>          version: 3.0
>          itemType: entity
>          item:
>            type: blah
>        - id: bar
>          version: 4.0
>          itemType: entity
>          item:
>         type: blah.blah
>
> Here, the "1.0" version is not used by anything; the auto-generated OSGi
> bundle would be version 2.0; catalog item foo would be 3.0; and catalog
> item bar would be 4.0.
>
> If we were starting from scratch, I'd be tempted to lock things down a
> lot more for what we actually support. For now, it feels like asking for
> an explicit version is a reasonable thing to do.
>
> ---
> We could also support the example below, where "1.0" is used by both the
> manifest and the item:
>
>     brooklyn.catalog:
>        version: 1.0
>        manifest:
>          symbolic_name: com.example.Foo
>        items:
>        - id: foo
>          itemType: entity
>          item:
>            type: blah
>
> The general rule would be: if there is a version in the manifest
> section, that will be used; if not then a top-level version number will
> be used. If that is also missing, then fail.
>
> i.e. we would not accept:
>
>     # Fails - manifest has no version.
>     brooklyn.catalog:
>        manifest:
>          symbolic_name: com.example.Foo
>        items:
>        - id: foo
>          version: 1.0
>          itemType: entity
>          item:
>            type: blah
>
> Thoughts?
>
> Aled
>
>
> On 02/03/2017 10:02, Geoff Macartney wrote:
> > I take Alex's point about the manifest being Java specific, and I agree
> > therefore we shouldn't insist on it.
> >
> > +1 also to preferring explicit name/version in the catalog.bom rather
> than
> > as API params, I agree we do
> > need to keep the version in source control.
> >
> > Question on your straw man, does the 'version' below
> >
> >      brooklyn.catalog:
> >         manifest:
> >           symbolic_name: com.example.Foo
> >           version: 1.0.0-SNAPSHOT
> >         items:
> >
> > _replace_ the 'version' in existing catalogs
> >
> > brooklyn.catalog:
> >      version: "0.1.0-SNAPSHOT"
> >
> > or is it an independently variable value, i.e. the version of the bundle,
> > separate from the version of catalog items that it happens to contain?
> > (Sounds a bit disquieting, but I guess it's possible).
> >
> > e.g. could you have
> >
> > brooklyn.catalog:
> >    version: 0.11.0-SNAPSHOT
> >    manifest:
> >      symbolic_name: com.example.Foo
> >      version: 1.0.0-SNAPSHOT
> >    items:
> >    - id: foo
> >      type: xyz
> >
> > which would be bundle com.example.Foo:1.0.0-SNAPSHOT, which happens to
> > contain item foo:0.11.0-SNAPSHOT?
> >
> > If so, what would happen if you left out the second line above? What
> > version of 'foo' would the catalog contain?
> >
> > Or does the version within the manifest _mean_ that it is not only the
> > bundle version you are specifying, but the catalog item versions too? (I
> > guess unless the item explicitly supplies its own version.)
> >
> > Geoff
> >
> >
> >
> >
> > On Wed, 1 Mar 2017 at 17:26 Aled Sage <al...@gmail.com> wrote:
> >
> > Hi all,
> >
> > Discussion of the REST api kicked off again in
> >
> https://github.com/apache/brooklyn-server/pull/485#issuecomment-283280366:
> >
> >      Alex wrote:
> >
> >      Requiring a MANIFEST.MF makes the input strongly java-centric; I'd
> >      like to appeal to people who write YAML blueprints with co-bundled
> >      scripts and images. They'd wonder why they can't simply make and
> >      upload a ZIP. They could probably be persuaded to supply a name and
> >      optional version on a CLI or UI, and understand why it is needed
> >      (hence supporting those args) but it would not be idiomatic to
> >      anyone but a java programmer to create a META-INF/ dir with a
> >      MANIFEST.MF and its syntax. Meanwhile it is very easy for us to
> >      convert the ZIP to a JAR on the server. Feels like uncontroversial
> >      good UX.
> >
> >      OTOH for a java programmer a MANIFEST.MF is natural, and they'd want
> >      to drop the name/version args if they are in that file, and I see no
> >      reason to forbid that pattern.
> >
> > I agree with Alex, that we should not require a META-INF/MANIFEST.MF. As
> > for Geoff's suggestion that we could auto-generate a manifest in the
> > `br` CLI, I'd prefer a more general solution that works for users of the
> > REST api as well (i.e. doing it server-side).
> >
> > ---
> > Svet suggested that the catalog.bom could give the symbolic name +
> > version, via additional metadata in that file.
> >
> > What I really like about that is it's in version control, becuase it's
> > in the file/repo. If alternatively the name/version are just passed as
> > REST api parameters, then it's not in version control (and is more
> > susceptible to typos etc).
> >
> > I'm not sure what we'd want this to look like. As a straw man:
> >
> >      brooklyn.catalog:
> >         manifest:
> >           symbolic_name: com.example.Foo
> >           version: 1.0.0-SNAPSHOT
> >         items:
> >         - ...
> >
> > If the user built their own real OSGi bundle, then they wouldn't need to
> > include this "manifest" section. If they did include that and it
> > contradicted the OSGi bundle's manifest, then we'd fail-fast.
> >
> > (Note this reminds me of the (unrelated) metadata described in
> > https://github.com/brooklyncentral/blueprint-repository- there is a
> > "publish" block that can be added to the catalog.bom.)
> >
> > ---
> > With Svet's suggestion, if there was no manifest section/file, then
> > should we auto-generate something from the item(s) in the catalog.bom?
> >
> > I can see how that could easily work for a .bom file that has a single
> > item (e.g. we use the catalog item's id + version, possibly with a
> > special prefix in the symbolic name to avoid conflicts).
> >
> > However, if there are multiple items then it would get trickier.
> >
> > I'm inclined to say that for a minimal viable product (MVP) we can
> > insist on the "manifest" section in the catalog.bom.
> >
> > Aled
> >
> >
> > On 20/12/2016 16:34, Svetoslav Neykov wrote:
> >>> Svet, if instead we tried to infer it from the catalog.bom, would we
> > require some additional metadata within the .bom file? Or would we use
> the
> > catalog item's id + version? I'm not convinced by the latter - it would
> > mean some .bom files would work and others wouldn't (e.g. if the .bom had
> > multiple items with different versions). Better to support the explicit
> > approach IMO.
> >> I imagine it would be additional metadata. On the other hand I don't see
> > a technical reason why we need an explicit symbolicName and version -
> they
> > can be auto-generated.
> >> Svet.
> >>
> >>
> >>> On 20.12.2016 г., at 17:50, Aled Sage <al...@gmail.com> wrote:
> >>>
> >>> Hi all,
> >>>
> >>> +1
> >>>
> >>> (D) sounds good. What version are you imagining the bundle would be, if
> > one runs `br catalog add ~/my/project/ --name com.example.myproject`?
> >>> ---
> >>> I like the idea of uploading a plain zip (rather than only supporting
> > OSGi bundles) - that makes it simpler for non-java folk. The use of OSGi
> > becomes a (hidden) implementation detail to many users.
> >>> ---
> >>> If auto-generating the manifest, I think we need the user to be
> explicit
> > about symbolic name and version. Having these supplied in the REST api
> call
> > (as Alex suggests) would achieve that.
> >>> Svet, if instead we tried to infer it from the catalog.bom, would we
> > require some additional metadata within the .bom file? Or would we use
> the
> > catalog item's id + version? I'm not convinced by the latter - it would
> > mean some .bom files would work and others wouldn't (e.g. if the .bom had
> > multiple items with different versions). Better to support the explicit
> > approach IMO.
> >>> ---
> >>> For E ("have a mechanism whereby deployed entities based on an affected
> > blueprint are optionally migrated to the new code"), that feels like a
> > separate discussion. It could equally apply to a pure YAML .bom file that
> > has been added to the catalog.
> >>> I suggest we discuss that in a separate email thread.
> >>>
> >>> ---
> >>> For (G), it's an interesting suggestion from Svet to make use of Karaf
> > Cellar for HA nodes. I'm hesitant (e.g. if restarting a standalone
> Brooklyn
> > node whose VM has died, then it adds big additional requirements for what
> > constitutes the "persisted state"). On the other hand, it's good to use
> > well-established technologies rather than re-inventing things!
> >>> An alternative ("pure brooklyn") approach could be to write the bundle
> > to persisted state; on rebind, we'd install + activate those bundles.
> >>> ---
> >>> For "catalogGroupId", I agree with Svet that in the initial use-case
> > this can be an implementation detail.
> >>> It could be set as the bundle's symbolic name + version: everything
> from
> > the bundle should be deleted at once, along with the bundle.
> >>> Longer term, I can see how exposing "catalogGroupId" to the user could
> > support more use-cases (e.g. for several catalog items from different
> > bundles to work together). I don't think we should try to support that
> yet.
> >>> Aled
> >>>
> >>>
> >>> On 19/12/2016 17:19, Geoff Macartney wrote:
> >>>> hi Alex,
> >>>>
> >>>> this looks like a good feature to have, I shall look at the PR as soon
> > as I
> >>>> can.
> >>>>
> >>>> The catalog.bom scanner feature was initially enabled by default, but
> we
> >>>> had to
> >>>> disable it because it turned out not to work properly with rebind. I
> > don't
> >>>> think
> >>>> it should be a lot of work to fix that but it hasn't been something
> > we've
> >>>> got round
> >>>> to yet.  This would be a great opportunity to look back at that.
> >>>>
> >>>> Some random thoughts:
> >>>>
> >>>> re (C), if we are going to treat the zips as bundles, my gut feel is
> > that
> >>>> we
> >>>> should insist on a manifest and get the metadata from it.  It doesn't
> > feel
> >>>> to me
> >>>> like it makes much sense to allow a zip file without a MANIFEST.MF but
> >>>> convey
> >>>> the intended bundle metadata to Brooklyn via HTTP headers.  And rather
> > than
> >>>> infer bundle metadata I think it's better to ask users to be explicit
> > about
> >>>> what
> >>>> their intentions are.  To make users lives easier, we could
> >>>> add a command to br to generate the manifest (locally) with correct
> > syntax,
> >>>> so that the manifest is in the right place, rather than have br add
> the
> > data
> >>>> to the "upload" request headers.
> >>>>
> >>>> re. (D) will be glad to have a look at it
> >>>>
> >>>> re. (E) it would certainly need to be optional - maybe keep it as an
> >>>> explicit
> >>>> separate command ('upgrade'?)
> >>>>
> >>>> (F) it does seem like a lot of work but might be nice for users who
> are
> > not
> >>>> keen on command lines.
> >>>>
> >>>> G - I:  we'll definitely need to pay close attention to persistence
> and
> >>>> rebind;
> >>>> I wonder also about HA operation, are there any additional
> implications?
> >>>>
> >>>> (J) I think it would be good to treat all the files from a jar, sorry
> >>>> bundle,
> >>>> as an atomic group - cleaner that way perhaps than allowing
> > delete/update
> >>>> of
> >>>> individual entries from a bundle on a piecemeal basis.  Rest support
> on
> >>>> delete
> >>>> catalog could warn about related catalog entries being deleted and ask
> > for
> >>>> a "--force" param to confirm.
> >>>>
> >>>> Geoff
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Fri, 16 Dec 2016 at 15:24 Svetoslav Neykov <
> >>>> svetoslav.neykov@cloudsoftcorp.com> wrote:
> >>>>
> >>>>> +1
> >>>>>
> >>>>> Some thoughts:
> >>>>>     * (A) add a utility class BundleMaker
> >>>>>       Sounds very similar to
> >>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles <
> >>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles>
> >>>>>       Looking at the code it's much more focused on zip files so I
> guess
> >>>>> there's no much overlap, but worth keeping in mind
> >>>>>     * (C) accept bundle symbolic name and version
> >>>>>       Why require them at all? Could infer them from the catalog.bom
> in
> > some
> >>>>> way - maybe require those properties to be in there. If not present
> are
> >>>>> they really needed?
> >>>>>     * (G) Bundles installed via this mechanism are not persisted
> > currently &
> >>>>> (I) We persist the individual catalog items as YAML, so we end up
> with
> > two
> >>>>> records
> >>>>>       Suggest marking the catalog items coming from bundles as
> >>>>> non-persistable. Then try to share the bundles between HA nodes.
> (Karaf
> >>>>> Cellar?)
> >>>>>     * (J) Introduce a catalogGroupId field on catalog items;
> >>>>>       Agree this could be useful and I like the idea of deleting the
> > bundle
> >>>>> altogether with the catalog items. From user's perspective I don't
> see
> > the
> >>>>> need for an extra field (i.e. it's an implementation detail).
> >>>>>
> >>>>> Svet.
> >>>>>
> >>>>>
> >>>>>> On 16.12.2016 г., at 12:50, Alex Heneveld <
> >>>>> alex.heneveld@cloudsoftcorp.com> wrote:
> >>>>>> Hi Brooklyners-
> >>>>>>
> >>>>>> In the code we currently have two routes for users to install new
> >>>>>> blueprints:
> >>>>>>
> >>>>>> (1) upload a catalog YAML file to /v1/catalog
> >>>>>>
> >>>>>> (2) install a bundle with catalog.bom in the root
> >>>>>>
> >>>>>> The feature (2) is disabled by default, but I'd like to move towards
> >>>>>> enabling it.  This will make it easier to create nicely structured
> BOM
> >>>>>> files because scripts etc can be taken out of the BOM, stored as
> > files in
> >>>>>> the same bundle.  (Because URLs of the form
> >>>>>> `classpath://scripts/install.sh`  use the bundle's classpath to
> > resolve.)
> >>>>>> As a first step in #485 [1] I do a few things:
> >>>>>>
> >>>>>> (A) add a utility class BundleMaker that lets us create and modify
> >>>>>> bundles/zips, to make it easier to do things we might want to with
> >>>>> bundles,
> >>>>>> especially for testing
> >>>>>>
> >>>>>> (B) add an endpoint to the REST API which allows uploading a bundle
> > ZIP
> >>>>>> (C) accept bundle symbolic name and version in that REST API to
> >>>>> facilitate
> >>>>>> uploading non-bundle ZIPs where the OSGi MANIFEST.MF is
> automatically
> >>>>>> generated
> >>>>>>
> >>>>>> With this PR, if you have a directory on your local file system with
> >>>>>> scripts and config files, and a BOM which refers to them, you can
> just
> >>>>> ZIP
> >>>>>> that up an upload it, specifying the bundle name so that a YAML
> > blueprint
> >>>>>> author never needs to touch any java-isms.
> >>>>>>
> >>>>>> Where I see this going is a development workflow where a user can
> edit
> >>>>>> files locally and upload the ZIP to have that installed, and if they
> > make
> >>>>>> changes locally they can POST it again to have catalog items updated
> >>>>>> (because default version is a SNAPSHOT).  We could also:
> >>>>>>
> >>>>>> (D) have `br catalog add ~/my/project/ --name my.project` create a
> ZIP
> >>>>> and
> >>>>>> POST it, with bundle name metadata, so essentially the user's
> process
> > is
> >>>>>> just to run that whenever they make a change
> >>>>>>
> >>>>>> (E) have a mechanism whereby deployed entities based on an affected
> >>>>>> blueprint are optionally migrated to the new code, so if you've
> > changed
> >>>>> an
> >>>>>> enricher the changes are picked up, or if say a launch.sh script has
> >>>>>> changed, a restart will run the new code
> >>>>>>
> >>>>>> The above are fairly straightforward programmatically (although good
> > user
> >>>>>> interaction with (E) needs some thought).  So I think we can pretty
> >>>>> quickly
> >>>>>> get to a much smoother dev workflow.
> >>>>>>
> >>>>>>
> >>>>>> That's the highlight of this message.  You can jump to the end,
> unless
> >>>>>> you're interested in some important but low level details...
> >>>>>>
> >>>>>>
> >>>>>> I'm also tempted by:
> >>>>>>
> >>>>>> (F) Integration with web-based IDE and/or Brooklyn reading and
> writing
> >>>>>> straight from GitHub -- but this seems like a lot of work and I'm
> not
> >>>>>> convinced it's much better than (D) workflow-wise
> >>>>>>
> >>>>>> Before we can change (2) to be the default, or start widely using
> the
> >>>>> POST
> >>>>>> a ZIP feature, we need to sort out some issues to do with
> persistence
> > and
> >>>>>> reloading:
> >>>>>>
> >>>>>> (G) Bundles installed via this mechanism are not persisted
> currently,
> > so
> >>>>> if
> >>>>>> you move to a different Brooklyn using the same backing store,
> you'll
> >>>>> lose
> >>>>>> those bundles
> >>>>>>
> >>>>>> (H) On rebind, bundles aren't always activated when needed, meaning
> > items
> >>>>>> can't be loaded
> >>>>>>
> >>>>>> (I) We persist the individual catalog items as YAML, so we end up
> with
> >>>>> two
> >>>>>> records — the YAML from the catalog.bom in the bundle, and the YAML
> >>>>>> persisted for the item.  This isn't a problem per se, but something
> to
> >>>>>> think about, and some sometimes surprising behaviour.  In particular
> > if
> >>>>> you
> >>>>>> delete the persisted YAML, the bundle is still there so the item is
> no
> >>>>>> longer deleted after a full rebind.
> >>>>>>
> >>>>>> One idea which might be useful is:
> >>>>>>
> >>>>>> (J) Introduce a catalogGroupId field on catalog items; this will do
> > two
> >>>>>> things:  if you try to delete an item with such a record, you'll be
> >>>>>> encouraged to delete all such items (maybe disallowed to delete an
> >>>>>> individual one), with the effect of deleting the bundle if it comes
> > from
> >>>>> a
> >>>>>> bundle; and when resolving types we search first for items with the
> > same
> >>>>>> catalogGroupId (so that e.g. if I install MyCluster:1.0 and
> > MyNode:1.0 in
> >>>>>> the same group, the former can refer simply to "MyNode" but if I
> > install
> >>>>> a
> >>>>>> 2.0 version of that group, the 1.0 cluster still loads the 1.0 node
> --
> >>>>> this
> >>>>>> has bitten people i the past)
> >>>>>>
> >>>>>> There is a related Brooklyn upgrade problem worth mentioning, which
> > the
> >>>>>> above might help with, where:
> >>>>>>
> >>>>>> (K) If I migrate from Brooklyn 10 to 11 when it comes out, I'll no
> > longer
> >>>>>> have certain entities that were at v10, since we don't include
> those;
> > an
> >>>>>> upgrade could include rules that certain groupIds need to be
> updated,
> > or
> >>>>> it
> >>>>>> can search and attempt to automatically apply the updates
> >>>>>>
> >>>>>>
> >>>>>> Quite a lot here and we don't need to solve it but I wanted to:
> >>>>>>
> >>>>>> * Share the current thinking
> >>>>>>
> >>>>>> * Get opinions on the general dev workflow suggested by (D)
> >>>>>>
> >>>>>>
> >>>>>> Thanks for feedback -- and if we like it help with (D) would be
> >>>>> appreciated!
> >>>>>> Best
> >>>>>> Alex
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> [1] . https://github.com/apache/brooklyn-server/pull/485
>
>

Re: Uploading ZIPs for a better dev workflow

[Geoff Macartney <ge...@cloudsoftcorp.com>]

Just on the last point of "We could also support the example below", I'd
say let's not even be that flexible - my feeling is that the more flexible
we are, the more confusing it is, and the harder to get right.  If we're
going to add this capability let's just have one way to do it; if you are
going to do a ZIP upload, you MUST have a catalog.bom, which MUST contain a
manifest section, which MUST have both a symbolic name and a version.  That
way everything's explicit and very clear, every time.



On Thu, 2 Mar 2017 at 11:45 Aled Sage <al...@gmail.com> wrote:

> Geoff, all,
>
> I was imagining the manifest version (in the catalog.bom) to be separate
> from the item version. The reason is that we support multiple items in
> the bom that can be independently versioned.
>
> Somone perverted could write:
>
>     brooklyn.catalog:
>        version: 1.0
>        manifest:
>          symbolic_name: com.example.Foo
>          version: 2.0
>        items:
>        - id: foo
>          version: 3.0
>          itemType: entity
>          item:
>            type: blah
>        - id: bar
>          version: 4.0
>          itemType: entity
>          item:
>         type: blah.blah
>
> Here, the "1.0" version is not used by anything; the auto-generated OSGi
> bundle would be version 2.0; catalog item foo would be 3.0; and catalog
> item bar would be 4.0.
>
> If we were starting from scratch, I'd be tempted to lock things down a
> lot more for what we actually support. For now, it feels like asking for
> an explicit version is a reasonable thing to do.
>
> ---
> We could also support the example below, where "1.0" is used by both the
> manifest and the item:
>
>     brooklyn.catalog:
>        version: 1.0
>        manifest:
>          symbolic_name: com.example.Foo
>        items:
>        - id: foo
>          itemType: entity
>          item:
>            type: blah
>
> The general rule would be: if there is a version in the manifest
> section, that will be used; if not then a top-level version number will
> be used. If that is also missing, then fail.
>
> i.e. we would not accept:
>
>     # Fails - manifest has no version.
>     brooklyn.catalog:
>        manifest:
>          symbolic_name: com.example.Foo
>        items:
>        - id: foo
>          version: 1.0
>          itemType: entity
>          item:
>            type: blah
>
> Thoughts?
>
> Aled
>
>
> On 02/03/2017 10:02, Geoff Macartney wrote:
> > I take Alex's point about the manifest being Java specific, and I agree
> > therefore we shouldn't insist on it.
> >
> > +1 also to preferring explicit name/version in the catalog.bom rather
> than
> > as API params, I agree we do
> > need to keep the version in source control.
> >
> > Question on your straw man, does the 'version' below
> >
> >      brooklyn.catalog:
> >         manifest:
> >           symbolic_name: com.example.Foo
> >           version: 1.0.0-SNAPSHOT
> >         items:
> >
> > _replace_ the 'version' in existing catalogs
> >
> > brooklyn.catalog:
> >      version: "0.1.0-SNAPSHOT"
> >
> > or is it an independently variable value, i.e. the version of the bundle,
> > separate from the version of catalog items that it happens to contain?
> > (Sounds a bit disquieting, but I guess it's possible).
> >
> > e.g. could you have
> >
> > brooklyn.catalog:
> >    version: 0.11.0-SNAPSHOT
> >    manifest:
> >      symbolic_name: com.example.Foo
> >      version: 1.0.0-SNAPSHOT
> >    items:
> >    - id: foo
> >      type: xyz
> >
> > which would be bundle com.example.Foo:1.0.0-SNAPSHOT, which happens to
> > contain item foo:0.11.0-SNAPSHOT?
> >
> > If so, what would happen if you left out the second line above? What
> > version of 'foo' would the catalog contain?
> >
> > Or does the version within the manifest _mean_ that it is not only the
> > bundle version you are specifying, but the catalog item versions too? (I
> > guess unless the item explicitly supplies its own version.)
> >
> > Geoff
> >
> >
> >
> >
> > On Wed, 1 Mar 2017 at 17:26 Aled Sage <al...@gmail.com> wrote:
> >
> > Hi all,
> >
> > Discussion of the REST api kicked off again in
> >
> https://github.com/apache/brooklyn-server/pull/485#issuecomment-283280366:
> >
> >      Alex wrote:
> >
> >      Requiring a MANIFEST.MF makes the input strongly java-centric; I'd
> >      like to appeal to people who write YAML blueprints with co-bundled
> >      scripts and images. They'd wonder why they can't simply make and
> >      upload a ZIP. They could probably be persuaded to supply a name and
> >      optional version on a CLI or UI, and understand why it is needed
> >      (hence supporting those args) but it would not be idiomatic to
> >      anyone but a java programmer to create a META-INF/ dir with a
> >      MANIFEST.MF and its syntax. Meanwhile it is very easy for us to
> >      convert the ZIP to a JAR on the server. Feels like uncontroversial
> >      good UX.
> >
> >      OTOH for a java programmer a MANIFEST.MF is natural, and they'd want
> >      to drop the name/version args if they are in that file, and I see no
> >      reason to forbid that pattern.
> >
> > I agree with Alex, that we should not require a META-INF/MANIFEST.MF. As
> > for Geoff's suggestion that we could auto-generate a manifest in the
> > `br` CLI, I'd prefer a more general solution that works for users of the
> > REST api as well (i.e. doing it server-side).
> >
> > ---
> > Svet suggested that the catalog.bom could give the symbolic name +
> > version, via additional metadata in that file.
> >
> > What I really like about that is it's in version control, becuase it's
> > in the file/repo. If alternatively the name/version are just passed as
> > REST api parameters, then it's not in version control (and is more
> > susceptible to typos etc).
> >
> > I'm not sure what we'd want this to look like. As a straw man:
> >
> >      brooklyn.catalog:
> >         manifest:
> >           symbolic_name: com.example.Foo
> >           version: 1.0.0-SNAPSHOT
> >         items:
> >         - ...
> >
> > If the user built their own real OSGi bundle, then they wouldn't need to
> > include this "manifest" section. If they did include that and it
> > contradicted the OSGi bundle's manifest, then we'd fail-fast.
> >
> > (Note this reminds me of the (unrelated) metadata described in
> > https://github.com/brooklyncentral/blueprint-repository- there is a
> > "publish" block that can be added to the catalog.bom.)
> >
> > ---
> > With Svet's suggestion, if there was no manifest section/file, then
> > should we auto-generate something from the item(s) in the catalog.bom?
> >
> > I can see how that could easily work for a .bom file that has a single
> > item (e.g. we use the catalog item's id + version, possibly with a
> > special prefix in the symbolic name to avoid conflicts).
> >
> > However, if there are multiple items then it would get trickier.
> >
> > I'm inclined to say that for a minimal viable product (MVP) we can
> > insist on the "manifest" section in the catalog.bom.
> >
> > Aled
> >
> >
> > On 20/12/2016 16:34, Svetoslav Neykov wrote:
> >>> Svet, if instead we tried to infer it from the catalog.bom, would we
> > require some additional metadata within the .bom file? Or would we use
> the
> > catalog item's id + version? I'm not convinced by the latter - it would
> > mean some .bom files would work and others wouldn't (e.g. if the .bom had
> > multiple items with different versions). Better to support the explicit
> > approach IMO.
> >> I imagine it would be additional metadata. On the other hand I don't see
> > a technical reason why we need an explicit symbolicName and version -
> they
> > can be auto-generated.
> >> Svet.
> >>
> >>
> >>> On 20.12.2016 г., at 17:50, Aled Sage <al...@gmail.com> wrote:
> >>>
> >>> Hi all,
> >>>
> >>> +1
> >>>
> >>> (D) sounds good. What version are you imagining the bundle would be, if
> > one runs `br catalog add ~/my/project/ --name com.example.myproject`?
> >>> ---
> >>> I like the idea of uploading a plain zip (rather than only supporting
> > OSGi bundles) - that makes it simpler for non-java folk. The use of OSGi
> > becomes a (hidden) implementation detail to many users.
> >>> ---
> >>> If auto-generating the manifest, I think we need the user to be
> explicit
> > about symbolic name and version. Having these supplied in the REST api
> call
> > (as Alex suggests) would achieve that.
> >>> Svet, if instead we tried to infer it from the catalog.bom, would we
> > require some additional metadata within the .bom file? Or would we use
> the
> > catalog item's id + version? I'm not convinced by the latter - it would
> > mean some .bom files would work and others wouldn't (e.g. if the .bom had
> > multiple items with different versions). Better to support the explicit
> > approach IMO.
> >>> ---
> >>> For E ("have a mechanism whereby deployed entities based on an affected
> > blueprint are optionally migrated to the new code"), that feels like a
> > separate discussion. It could equally apply to a pure YAML .bom file that
> > has been added to the catalog.
> >>> I suggest we discuss that in a separate email thread.
> >>>
> >>> ---
> >>> For (G), it's an interesting suggestion from Svet to make use of Karaf
> > Cellar for HA nodes. I'm hesitant (e.g. if restarting a standalone
> Brooklyn
> > node whose VM has died, then it adds big additional requirements for what
> > constitutes the "persisted state"). On the other hand, it's good to use
> > well-established technologies rather than re-inventing things!
> >>> An alternative ("pure brooklyn") approach could be to write the bundle
> > to persisted state; on rebind, we'd install + activate those bundles.
> >>> ---
> >>> For "catalogGroupId", I agree with Svet that in the initial use-case
> > this can be an implementation detail.
> >>> It could be set as the bundle's symbolic name + version: everything
> from
> > the bundle should be deleted at once, along with the bundle.
> >>> Longer term, I can see how exposing "catalogGroupId" to the user could
> > support more use-cases (e.g. for several catalog items from different
> > bundles to work together). I don't think we should try to support that
> yet.
> >>> Aled
> >>>
> >>>
> >>> On 19/12/2016 17:19, Geoff Macartney wrote:
> >>>> hi Alex,
> >>>>
> >>>> this looks like a good feature to have, I shall look at the PR as soon
> > as I
> >>>> can.
> >>>>
> >>>> The catalog.bom scanner feature was initially enabled by default, but
> we
> >>>> had to
> >>>> disable it because it turned out not to work properly with rebind. I
> > don't
> >>>> think
> >>>> it should be a lot of work to fix that but it hasn't been something
> > we've
> >>>> got round
> >>>> to yet.  This would be a great opportunity to look back at that.
> >>>>
> >>>> Some random thoughts:
> >>>>
> >>>> re (C), if we are going to treat the zips as bundles, my gut feel is
> > that
> >>>> we
> >>>> should insist on a manifest and get the metadata from it.  It doesn't
> > feel
> >>>> to me
> >>>> like it makes much sense to allow a zip file without a MANIFEST.MF but
> >>>> convey
> >>>> the intended bundle metadata to Brooklyn via HTTP headers.  And rather
> > than
> >>>> infer bundle metadata I think it's better to ask users to be explicit
> > about
> >>>> what
> >>>> their intentions are.  To make users lives easier, we could
> >>>> add a command to br to generate the manifest (locally) with correct
> > syntax,
> >>>> so that the manifest is in the right place, rather than have br add
> the
> > data
> >>>> to the "upload" request headers.
> >>>>
> >>>> re. (D) will be glad to have a look at it
> >>>>
> >>>> re. (E) it would certainly need to be optional - maybe keep it as an
> >>>> explicit
> >>>> separate command ('upgrade'?)
> >>>>
> >>>> (F) it does seem like a lot of work but might be nice for users who
> are
> > not
> >>>> keen on command lines.
> >>>>
> >>>> G - I:  we'll definitely need to pay close attention to persistence
> and
> >>>> rebind;
> >>>> I wonder also about HA operation, are there any additional
> implications?
> >>>>
> >>>> (J) I think it would be good to treat all the files from a jar, sorry
> >>>> bundle,
> >>>> as an atomic group - cleaner that way perhaps than allowing
> > delete/update
> >>>> of
> >>>> individual entries from a bundle on a piecemeal basis.  Rest support
> on
> >>>> delete
> >>>> catalog could warn about related catalog entries being deleted and ask
> > for
> >>>> a "--force" param to confirm.
> >>>>
> >>>> Geoff
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Fri, 16 Dec 2016 at 15:24 Svetoslav Neykov <
> >>>> svetoslav.neykov@cloudsoftcorp.com> wrote:
> >>>>
> >>>>> +1
> >>>>>
> >>>>> Some thoughts:
> >>>>>     * (A) add a utility class BundleMaker
> >>>>>       Sounds very similar to
> >>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles <
> >>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles>
> >>>>>       Looking at the code it's much more focused on zip files so I
> guess
> >>>>> there's no much overlap, but worth keeping in mind
> >>>>>     * (C) accept bundle symbolic name and version
> >>>>>       Why require them at all? Could infer them from the catalog.bom
> in
> > some
> >>>>> way - maybe require those properties to be in there. If not present
> are
> >>>>> they really needed?
> >>>>>     * (G) Bundles installed via this mechanism are not persisted
> > currently &
> >>>>> (I) We persist the individual catalog items as YAML, so we end up
> with
> > two
> >>>>> records
> >>>>>       Suggest marking the catalog items coming from bundles as
> >>>>> non-persistable. Then try to share the bundles between HA nodes.
> (Karaf
> >>>>> Cellar?)
> >>>>>     * (J) Introduce a catalogGroupId field on catalog items;
> >>>>>       Agree this could be useful and I like the idea of deleting the
> > bundle
> >>>>> altogether with the catalog items. From user's perspective I don't
> see
> > the
> >>>>> need for an extra field (i.e. it's an implementation detail).
> >>>>>
> >>>>> Svet.
> >>>>>
> >>>>>
> >>>>>> On 16.12.2016 г., at 12:50, Alex Heneveld <
> >>>>> alex.heneveld@cloudsoftcorp.com> wrote:
> >>>>>> Hi Brooklyners-
> >>>>>>
> >>>>>> In the code we currently have two routes for users to install new
> >>>>>> blueprints:
> >>>>>>
> >>>>>> (1) upload a catalog YAML file to /v1/catalog
> >>>>>>
> >>>>>> (2) install a bundle with catalog.bom in the root
> >>>>>>
> >>>>>> The feature (2) is disabled by default, but I'd like to move towards
> >>>>>> enabling it.  This will make it easier to create nicely structured
> BOM
> >>>>>> files because scripts etc can be taken out of the BOM, stored as
> > files in
> >>>>>> the same bundle.  (Because URLs of the form
> >>>>>> `classpath://scripts/install.sh`  use the bundle's classpath to
> > resolve.)
> >>>>>> As a first step in #485 [1] I do a few things:
> >>>>>>
> >>>>>> (A) add a utility class BundleMaker that lets us create and modify
> >>>>>> bundles/zips, to make it easier to do things we might want to with
> >>>>> bundles,
> >>>>>> especially for testing
> >>>>>>
> >>>>>> (B) add an endpoint to the REST API which allows uploading a bundle
> > ZIP
> >>>>>> (C) accept bundle symbolic name and version in that REST API to
> >>>>> facilitate
> >>>>>> uploading non-bundle ZIPs where the OSGi MANIFEST.MF is
> automatically
> >>>>>> generated
> >>>>>>
> >>>>>> With this PR, if you have a directory on your local file system with
> >>>>>> scripts and config files, and a BOM which refers to them, you can
> just
> >>>>> ZIP
> >>>>>> that up an upload it, specifying the bundle name so that a YAML
> > blueprint
> >>>>>> author never needs to touch any java-isms.
> >>>>>>
> >>>>>> Where I see this going is a development workflow where a user can
> edit
> >>>>>> files locally and upload the ZIP to have that installed, and if they
> > make
> >>>>>> changes locally they can POST it again to have catalog items updated
> >>>>>> (because default version is a SNAPSHOT).  We could also:
> >>>>>>
> >>>>>> (D) have `br catalog add ~/my/project/ --name my.project` create a
> ZIP
> >>>>> and
> >>>>>> POST it, with bundle name metadata, so essentially the user's
> process
> > is
> >>>>>> just to run that whenever they make a change
> >>>>>>
> >>>>>> (E) have a mechanism whereby deployed entities based on an affected
> >>>>>> blueprint are optionally migrated to the new code, so if you've
> > changed
> >>>>> an
> >>>>>> enricher the changes are picked up, or if say a launch.sh script has
> >>>>>> changed, a restart will run the new code
> >>>>>>
> >>>>>> The above are fairly straightforward programmatically (although good
> > user
> >>>>>> interaction with (E) needs some thought).  So I think we can pretty
> >>>>> quickly
> >>>>>> get to a much smoother dev workflow.
> >>>>>>
> >>>>>>
> >>>>>> That's the highlight of this message.  You can jump to the end,
> unless
> >>>>>> you're interested in some important but low level details...
> >>>>>>
> >>>>>>
> >>>>>> I'm also tempted by:
> >>>>>>
> >>>>>> (F) Integration with web-based IDE and/or Brooklyn reading and
> writing
> >>>>>> straight from GitHub -- but this seems like a lot of work and I'm
> not
> >>>>>> convinced it's much better than (D) workflow-wise
> >>>>>>
> >>>>>> Before we can change (2) to be the default, or start widely using
> the
> >>>>> POST
> >>>>>> a ZIP feature, we need to sort out some issues to do with
> persistence
> > and
> >>>>>> reloading:
> >>>>>>
> >>>>>> (G) Bundles installed via this mechanism are not persisted
> currently,
> > so
> >>>>> if
> >>>>>> you move to a different Brooklyn using the same backing store,
> you'll
> >>>>> lose
> >>>>>> those bundles
> >>>>>>
> >>>>>> (H) On rebind, bundles aren't always activated when needed, meaning
> > items
> >>>>>> can't be loaded
> >>>>>>
> >>>>>> (I) We persist the individual catalog items as YAML, so we end up
> with
> >>>>> two
> >>>>>> records — the YAML from the catalog.bom in the bundle, and the YAML
> >>>>>> persisted for the item.  This isn't a problem per se, but something
> to
> >>>>>> think about, and some sometimes surprising behaviour.  In particular
> > if
> >>>>> you
> >>>>>> delete the persisted YAML, the bundle is still there so the item is
> no
> >>>>>> longer deleted after a full rebind.
> >>>>>>
> >>>>>> One idea which might be useful is:
> >>>>>>
> >>>>>> (J) Introduce a catalogGroupId field on catalog items; this will do
> > two
> >>>>>> things:  if you try to delete an item with such a record, you'll be
> >>>>>> encouraged to delete all such items (maybe disallowed to delete an
> >>>>>> individual one), with the effect of deleting the bundle if it comes
> > from
> >>>>> a
> >>>>>> bundle; and when resolving types we search first for items with the
> > same
> >>>>>> catalogGroupId (so that e.g. if I install MyCluster:1.0 and
> > MyNode:1.0 in
> >>>>>> the same group, the former can refer simply to "MyNode" but if I
> > install
> >>>>> a
> >>>>>> 2.0 version of that group, the 1.0 cluster still loads the 1.0 node
> --
> >>>>> this
> >>>>>> has bitten people i the past)
> >>>>>>
> >>>>>> There is a related Brooklyn upgrade problem worth mentioning, which
> > the
> >>>>>> above might help with, where:
> >>>>>>
> >>>>>> (K) If I migrate from Brooklyn 10 to 11 when it comes out, I'll no
> > longer
> >>>>>> have certain entities that were at v10, since we don't include
> those;
> > an
> >>>>>> upgrade could include rules that certain groupIds need to be
> updated,
> > or
> >>>>> it
> >>>>>> can search and attempt to automatically apply the updates
> >>>>>>
> >>>>>>
> >>>>>> Quite a lot here and we don't need to solve it but I wanted to:
> >>>>>>
> >>>>>> * Share the current thinking
> >>>>>>
> >>>>>> * Get opinions on the general dev workflow suggested by (D)
> >>>>>>
> >>>>>>
> >>>>>> Thanks for feedback -- and if we like it help with (D) would be
> >>>>> appreciated!
> >>>>>> Best
> >>>>>> Alex
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> [1] . https://github.com/apache/brooklyn-server/pull/485
>
>

Re: Uploading ZIPs for a better dev workflow

[Aled Sage <al...@gmail.com>]

Geoff, all,

I was imagining the manifest version (in the catalog.bom) to be separate 
from the item version. The reason is that we support multiple items in 
the bom that can be independently versioned.

Somone perverted could write:

    brooklyn.catalog:
       version: 1.0
       manifest:
         symbolic_name: com.example.Foo
         version: 2.0
       items:
       - id: foo
         version: 3.0
         itemType: entity
         item:
           type: blah
       - id: bar
         version: 4.0
         itemType: entity
         item:
        type: blah.blah

Here, the "1.0" version is not used by anything; the auto-generated OSGi 
bundle would be version 2.0; catalog item foo would be 3.0; and catalog 
item bar would be 4.0.

If we were starting from scratch, I'd be tempted to lock things down a 
lot more for what we actually support. For now, it feels like asking for 
an explicit version is a reasonable thing to do.

---
We could also support the example below, where "1.0" is used by both the 
manifest and the item:

    brooklyn.catalog:
       version: 1.0
       manifest:
         symbolic_name: com.example.Foo
       items:
       - id: foo
         itemType: entity
         item:
           type: blah

The general rule would be: if there is a version in the manifest 
section, that will be used; if not then a top-level version number will 
be used. If that is also missing, then fail.

i.e. we would not accept:

    # Fails - manifest has no version.
    brooklyn.catalog:
       manifest:
         symbolic_name: com.example.Foo
       items:
       - id: foo
         version: 1.0
         itemType: entity
         item:
           type: blah

Thoughts?

Aled


On 02/03/2017 10:02, Geoff Macartney wrote:
> I take Alex's point about the manifest being Java specific, and I agree
> therefore we shouldn't insist on it.
>
> +1 also to preferring explicit name/version in the catalog.bom rather than
> as API params, I agree we do
> need to keep the version in source control.
>
> Question on your straw man, does the 'version' below
>
>      brooklyn.catalog:
>         manifest:
>           symbolic_name: com.example.Foo
>           version: 1.0.0-SNAPSHOT
>         items:
>
> _replace_ the 'version' in existing catalogs
>
> brooklyn.catalog:
>      version: "0.1.0-SNAPSHOT"
>
> or is it an independently variable value, i.e. the version of the bundle,
> separate from the version of catalog items that it happens to contain?
> (Sounds a bit disquieting, but I guess it's possible).
>
> e.g. could you have
>
> brooklyn.catalog:
>    version: 0.11.0-SNAPSHOT
>    manifest:
>      symbolic_name: com.example.Foo
>      version: 1.0.0-SNAPSHOT
>    items:
>    - id: foo
>      type: xyz
>
> which would be bundle com.example.Foo:1.0.0-SNAPSHOT, which happens to
> contain item foo:0.11.0-SNAPSHOT?
>
> If so, what would happen if you left out the second line above? What
> version of 'foo' would the catalog contain?
>
> Or does the version within the manifest _mean_ that it is not only the
> bundle version you are specifying, but the catalog item versions too? (I
> guess unless the item explicitly supplies its own version.)
>
> Geoff
>
>
>
>
> On Wed, 1 Mar 2017 at 17:26 Aled Sage <al...@gmail.com> wrote:
>
> Hi all,
>
> Discussion of the REST api kicked off again in
> https://github.com/apache/brooklyn-server/pull/485#issuecomment-283280366:
>
>      Alex wrote:
>
>      Requiring a MANIFEST.MF makes the input strongly java-centric; I'd
>      like to appeal to people who write YAML blueprints with co-bundled
>      scripts and images. They'd wonder why they can't simply make and
>      upload a ZIP. They could probably be persuaded to supply a name and
>      optional version on a CLI or UI, and understand why it is needed
>      (hence supporting those args) but it would not be idiomatic to
>      anyone but a java programmer to create a META-INF/ dir with a
>      MANIFEST.MF and its syntax. Meanwhile it is very easy for us to
>      convert the ZIP to a JAR on the server. Feels like uncontroversial
>      good UX.
>
>      OTOH for a java programmer a MANIFEST.MF is natural, and they'd want
>      to drop the name/version args if they are in that file, and I see no
>      reason to forbid that pattern.
>
> I agree with Alex, that we should not require a META-INF/MANIFEST.MF. As
> for Geoff's suggestion that we could auto-generate a manifest in the
> `br` CLI, I'd prefer a more general solution that works for users of the
> REST api as well (i.e. doing it server-side).
>
> ---
> Svet suggested that the catalog.bom could give the symbolic name +
> version, via additional metadata in that file.
>
> What I really like about that is it's in version control, becuase it's
> in the file/repo. If alternatively the name/version are just passed as
> REST api parameters, then it's not in version control (and is more
> susceptible to typos etc).
>
> I'm not sure what we'd want this to look like. As a straw man:
>
>      brooklyn.catalog:
>         manifest:
>           symbolic_name: com.example.Foo
>           version: 1.0.0-SNAPSHOT
>         items:
>         - ...
>
> If the user built their own real OSGi bundle, then they wouldn't need to
> include this "manifest" section. If they did include that and it
> contradicted the OSGi bundle's manifest, then we'd fail-fast.
>
> (Note this reminds me of the (unrelated) metadata described in
> https://github.com/brooklyncentral/blueprint-repository- there is a
> "publish" block that can be added to the catalog.bom.)
>
> ---
> With Svet's suggestion, if there was no manifest section/file, then
> should we auto-generate something from the item(s) in the catalog.bom?
>
> I can see how that could easily work for a .bom file that has a single
> item (e.g. we use the catalog item's id + version, possibly with a
> special prefix in the symbolic name to avoid conflicts).
>
> However, if there are multiple items then it would get trickier.
>
> I'm inclined to say that for a minimal viable product (MVP) we can
> insist on the "manifest" section in the catalog.bom.
>
> Aled
>
>
> On 20/12/2016 16:34, Svetoslav Neykov wrote:
>>> Svet, if instead we tried to infer it from the catalog.bom, would we
> require some additional metadata within the .bom file? Or would we use the
> catalog item's id + version? I'm not convinced by the latter - it would
> mean some .bom files would work and others wouldn't (e.g. if the .bom had
> multiple items with different versions). Better to support the explicit
> approach IMO.
>> I imagine it would be additional metadata. On the other hand I don't see
> a technical reason why we need an explicit symbolicName and version - they
> can be auto-generated.
>> Svet.
>>
>>
>>> On 20.12.2016 \u0433., at 17:50, Aled Sage <al...@gmail.com> wrote:
>>>
>>> Hi all,
>>>
>>> +1
>>>
>>> (D) sounds good. What version are you imagining the bundle would be, if
> one runs `br catalog add ~/my/project/ --name com.example.myproject`?
>>> ---
>>> I like the idea of uploading a plain zip (rather than only supporting
> OSGi bundles) - that makes it simpler for non-java folk. The use of OSGi
> becomes a (hidden) implementation detail to many users.
>>> ---
>>> If auto-generating the manifest, I think we need the user to be explicit
> about symbolic name and version. Having these supplied in the REST api call
> (as Alex suggests) would achieve that.
>>> Svet, if instead we tried to infer it from the catalog.bom, would we
> require some additional metadata within the .bom file? Or would we use the
> catalog item's id + version? I'm not convinced by the latter - it would
> mean some .bom files would work and others wouldn't (e.g. if the .bom had
> multiple items with different versions). Better to support the explicit
> approach IMO.
>>> ---
>>> For E ("have a mechanism whereby deployed entities based on an affected
> blueprint are optionally migrated to the new code"), that feels like a
> separate discussion. It could equally apply to a pure YAML .bom file that
> has been added to the catalog.
>>> I suggest we discuss that in a separate email thread.
>>>
>>> ---
>>> For (G), it's an interesting suggestion from Svet to make use of Karaf
> Cellar for HA nodes. I'm hesitant (e.g. if restarting a standalone Brooklyn
> node whose VM has died, then it adds big additional requirements for what
> constitutes the "persisted state"). On the other hand, it's good to use
> well-established technologies rather than re-inventing things!
>>> An alternative ("pure brooklyn") approach could be to write the bundle
> to persisted state; on rebind, we'd install + activate those bundles.
>>> ---
>>> For "catalogGroupId", I agree with Svet that in the initial use-case
> this can be an implementation detail.
>>> It could be set as the bundle's symbolic name + version: everything from
> the bundle should be deleted at once, along with the bundle.
>>> Longer term, I can see how exposing "catalogGroupId" to the user could
> support more use-cases (e.g. for several catalog items from different
> bundles to work together). I don't think we should try to support that yet.
>>> Aled
>>>
>>>
>>> On 19/12/2016 17:19, Geoff Macartney wrote:
>>>> hi Alex,
>>>>
>>>> this looks like a good feature to have, I shall look at the PR as soon
> as I
>>>> can.
>>>>
>>>> The catalog.bom scanner feature was initially enabled by default, but we
>>>> had to
>>>> disable it because it turned out not to work properly with rebind. I
> don't
>>>> think
>>>> it should be a lot of work to fix that but it hasn't been something
> we've
>>>> got round
>>>> to yet.  This would be a great opportunity to look back at that.
>>>>
>>>> Some random thoughts:
>>>>
>>>> re (C), if we are going to treat the zips as bundles, my gut feel is
> that
>>>> we
>>>> should insist on a manifest and get the metadata from it.  It doesn't
> feel
>>>> to me
>>>> like it makes much sense to allow a zip file without a MANIFEST.MF but
>>>> convey
>>>> the intended bundle metadata to Brooklyn via HTTP headers.  And rather
> than
>>>> infer bundle metadata I think it's better to ask users to be explicit
> about
>>>> what
>>>> their intentions are.  To make users lives easier, we could
>>>> add a command to br to generate the manifest (locally) with correct
> syntax,
>>>> so that the manifest is in the right place, rather than have br add the
> data
>>>> to the "upload" request headers.
>>>>
>>>> re. (D) will be glad to have a look at it
>>>>
>>>> re. (E) it would certainly need to be optional - maybe keep it as an
>>>> explicit
>>>> separate command ('upgrade'?)
>>>>
>>>> (F) it does seem like a lot of work but might be nice for users who are
> not
>>>> keen on command lines.
>>>>
>>>> G - I:  we'll definitely need to pay close attention to persistence and
>>>> rebind;
>>>> I wonder also about HA operation, are there any additional implications?
>>>>
>>>> (J) I think it would be good to treat all the files from a jar, sorry
>>>> bundle,
>>>> as an atomic group - cleaner that way perhaps than allowing
> delete/update
>>>> of
>>>> individual entries from a bundle on a piecemeal basis.  Rest support on
>>>> delete
>>>> catalog could warn about related catalog entries being deleted and ask
> for
>>>> a "--force" param to confirm.
>>>>
>>>> Geoff
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, 16 Dec 2016 at 15:24 Svetoslav Neykov <
>>>> svetoslav.neykov@cloudsoftcorp.com> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> Some thoughts:
>>>>>     * (A) add a utility class BundleMaker
>>>>>       Sounds very similar to
>>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles <
>>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles>
>>>>>       Looking at the code it's much more focused on zip files so I guess
>>>>> there's no much overlap, but worth keeping in mind
>>>>>     * (C) accept bundle symbolic name and version
>>>>>       Why require them at all? Could infer them from the catalog.bom in
> some
>>>>> way - maybe require those properties to be in there. If not present are
>>>>> they really needed?
>>>>>     * (G) Bundles installed via this mechanism are not persisted
> currently &
>>>>> (I) We persist the individual catalog items as YAML, so we end up with
> two
>>>>> records
>>>>>       Suggest marking the catalog items coming from bundles as
>>>>> non-persistable. Then try to share the bundles between HA nodes. (Karaf
>>>>> Cellar?)
>>>>>     * (J) Introduce a catalogGroupId field on catalog items;
>>>>>       Agree this could be useful and I like the idea of deleting the
> bundle
>>>>> altogether with the catalog items. From user's perspective I don't see
> the
>>>>> need for an extra field (i.e. it's an implementation detail).
>>>>>
>>>>> Svet.
>>>>>
>>>>>
>>>>>> On 16.12.2016 \u0433., at 12:50, Alex Heneveld <
>>>>> alex.heneveld@cloudsoftcorp.com> wrote:
>>>>>> Hi Brooklyners-
>>>>>>
>>>>>> In the code we currently have two routes for users to install new
>>>>>> blueprints:
>>>>>>
>>>>>> (1) upload a catalog YAML file to /v1/catalog
>>>>>>
>>>>>> (2) install a bundle with catalog.bom in the root
>>>>>>
>>>>>> The feature (2) is disabled by default, but I'd like to move towards
>>>>>> enabling it.  This will make it easier to create nicely structured BOM
>>>>>> files because scripts etc can be taken out of the BOM, stored as
> files in
>>>>>> the same bundle.  (Because URLs of the form
>>>>>> `classpath://scripts/install.sh`  use the bundle's classpath to
> resolve.)
>>>>>> As a first step in #485 [1] I do a few things:
>>>>>>
>>>>>> (A) add a utility class BundleMaker that lets us create and modify
>>>>>> bundles/zips, to make it easier to do things we might want to with
>>>>> bundles,
>>>>>> especially for testing
>>>>>>
>>>>>> (B) add an endpoint to the REST API which allows uploading a bundle
> ZIP
>>>>>> (C) accept bundle symbolic name and version in that REST API to
>>>>> facilitate
>>>>>> uploading non-bundle ZIPs where the OSGi MANIFEST.MF is automatically
>>>>>> generated
>>>>>>
>>>>>> With this PR, if you have a directory on your local file system with
>>>>>> scripts and config files, and a BOM which refers to them, you can just
>>>>> ZIP
>>>>>> that up an upload it, specifying the bundle name so that a YAML
> blueprint
>>>>>> author never needs to touch any java-isms.
>>>>>>
>>>>>> Where I see this going is a development workflow where a user can edit
>>>>>> files locally and upload the ZIP to have that installed, and if they
> make
>>>>>> changes locally they can POST it again to have catalog items updated
>>>>>> (because default version is a SNAPSHOT).  We could also:
>>>>>>
>>>>>> (D) have `br catalog add ~/my/project/ --name my.project` create a ZIP
>>>>> and
>>>>>> POST it, with bundle name metadata, so essentially the user's process
> is
>>>>>> just to run that whenever they make a change
>>>>>>
>>>>>> (E) have a mechanism whereby deployed entities based on an affected
>>>>>> blueprint are optionally migrated to the new code, so if you've
> changed
>>>>> an
>>>>>> enricher the changes are picked up, or if say a launch.sh script has
>>>>>> changed, a restart will run the new code
>>>>>>
>>>>>> The above are fairly straightforward programmatically (although good
> user
>>>>>> interaction with (E) needs some thought).  So I think we can pretty
>>>>> quickly
>>>>>> get to a much smoother dev workflow.
>>>>>>
>>>>>>
>>>>>> That's the highlight of this message.  You can jump to the end, unless
>>>>>> you're interested in some important but low level details...
>>>>>>
>>>>>>
>>>>>> I'm also tempted by:
>>>>>>
>>>>>> (F) Integration with web-based IDE and/or Brooklyn reading and writing
>>>>>> straight from GitHub -- but this seems like a lot of work and I'm not
>>>>>> convinced it's much better than (D) workflow-wise
>>>>>>
>>>>>> Before we can change (2) to be the default, or start widely using the
>>>>> POST
>>>>>> a ZIP feature, we need to sort out some issues to do with persistence
> and
>>>>>> reloading:
>>>>>>
>>>>>> (G) Bundles installed via this mechanism are not persisted currently,
> so
>>>>> if
>>>>>> you move to a different Brooklyn using the same backing store, you'll
>>>>> lose
>>>>>> those bundles
>>>>>>
>>>>>> (H) On rebind, bundles aren't always activated when needed, meaning
> items
>>>>>> can't be loaded
>>>>>>
>>>>>> (I) We persist the individual catalog items as YAML, so we end up with
>>>>> two
>>>>>> records \u2014 the YAML from the catalog.bom in the bundle, and the YAML
>>>>>> persisted for the item.  This isn't a problem per se, but something to
>>>>>> think about, and some sometimes surprising behaviour.  In particular
> if
>>>>> you
>>>>>> delete the persisted YAML, the bundle is still there so the item is no
>>>>>> longer deleted after a full rebind.
>>>>>>
>>>>>> One idea which might be useful is:
>>>>>>
>>>>>> (J) Introduce a catalogGroupId field on catalog items; this will do
> two
>>>>>> things:  if you try to delete an item with such a record, you'll be
>>>>>> encouraged to delete all such items (maybe disallowed to delete an
>>>>>> individual one), with the effect of deleting the bundle if it comes
> from
>>>>> a
>>>>>> bundle; and when resolving types we search first for items with the
> same
>>>>>> catalogGroupId (so that e.g. if I install MyCluster:1.0 and
> MyNode:1.0 in
>>>>>> the same group, the former can refer simply to "MyNode" but if I
> install
>>>>> a
>>>>>> 2.0 version of that group, the 1.0 cluster still loads the 1.0 node --
>>>>> this
>>>>>> has bitten people i the past)
>>>>>>
>>>>>> There is a related Brooklyn upgrade problem worth mentioning, which
> the
>>>>>> above might help with, where:
>>>>>>
>>>>>> (K) If I migrate from Brooklyn 10 to 11 when it comes out, I'll no
> longer
>>>>>> have certain entities that were at v10, since we don't include those;
> an
>>>>>> upgrade could include rules that certain groupIds need to be updated,
> or
>>>>> it
>>>>>> can search and attempt to automatically apply the updates
>>>>>>
>>>>>>
>>>>>> Quite a lot here and we don't need to solve it but I wanted to:
>>>>>>
>>>>>> * Share the current thinking
>>>>>>
>>>>>> * Get opinions on the general dev workflow suggested by (D)
>>>>>>
>>>>>>
>>>>>> Thanks for feedback -- and if we like it help with (D) would be
>>>>> appreciated!
>>>>>> Best
>>>>>> Alex
>>>>>>
>>>>>>
>>>>>>
>>>>>> [1] . https://github.com/apache/brooklyn-server/pull/485

Re: Uploading ZIPs for a better dev workflow

[Geoff Macartney <ge...@cloudsoftcorp.com>]

I take Alex's point about the manifest being Java specific, and I agree
therefore we shouldn't insist on it.

+1 also to preferring explicit name/version in the catalog.bom rather than
as API params, I agree we do
need to keep the version in source control.

Question on your straw man, does the 'version' below

    brooklyn.catalog:
       manifest:
         symbolic_name: com.example.Foo
         version: 1.0.0-SNAPSHOT
       items:

_replace_ the 'version' in existing catalogs

brooklyn.catalog:
    version: "0.1.0-SNAPSHOT"

or is it an independently variable value, i.e. the version of the bundle,
separate from the version of catalog items that it happens to contain?
(Sounds a bit disquieting, but I guess it's possible).

e.g. could you have

brooklyn.catalog:
  version: 0.11.0-SNAPSHOT
  manifest:
    symbolic_name: com.example.Foo
    version: 1.0.0-SNAPSHOT
  items:
  - id: foo
    type: xyz

which would be bundle com.example.Foo:1.0.0-SNAPSHOT, which happens to
contain item foo:0.11.0-SNAPSHOT?

If so, what would happen if you left out the second line above? What
version of 'foo' would the catalog contain?

Or does the version within the manifest _mean_ that it is not only the
bundle version you are specifying, but the catalog item versions too? (I
guess unless the item explicitly supplies its own version.)

Geoff




On Wed, 1 Mar 2017 at 17:26 Aled Sage <al...@gmail.com> wrote:

Hi all,

Discussion of the REST api kicked off again in
https://github.com/apache/brooklyn-server/pull/485#issuecomment-283280366:

    Alex wrote:

    Requiring a MANIFEST.MF makes the input strongly java-centric; I'd
    like to appeal to people who write YAML blueprints with co-bundled
    scripts and images. They'd wonder why they can't simply make and
    upload a ZIP. They could probably be persuaded to supply a name and
    optional version on a CLI or UI, and understand why it is needed
    (hence supporting those args) but it would not be idiomatic to
    anyone but a java programmer to create a META-INF/ dir with a
    MANIFEST.MF and its syntax. Meanwhile it is very easy for us to
    convert the ZIP to a JAR on the server. Feels like uncontroversial
    good UX.

    OTOH for a java programmer a MANIFEST.MF is natural, and they'd want
    to drop the name/version args if they are in that file, and I see no
    reason to forbid that pattern.

I agree with Alex, that we should not require a META-INF/MANIFEST.MF. As
for Geoff's suggestion that we could auto-generate a manifest in the
`br` CLI, I'd prefer a more general solution that works for users of the
REST api as well (i.e. doing it server-side).

---
Svet suggested that the catalog.bom could give the symbolic name +
version, via additional metadata in that file.

What I really like about that is it's in version control, becuase it's
in the file/repo. If alternatively the name/version are just passed as
REST api parameters, then it's not in version control (and is more
susceptible to typos etc).

I'm not sure what we'd want this to look like. As a straw man:

    brooklyn.catalog:
       manifest:
         symbolic_name: com.example.Foo
         version: 1.0.0-SNAPSHOT
       items:
       - ...

If the user built their own real OSGi bundle, then they wouldn't need to
include this "manifest" section. If they did include that and it
contradicted the OSGi bundle's manifest, then we'd fail-fast.

(Note this reminds me of the (unrelated) metadata described in
https://github.com/brooklyncentral/blueprint-repository- there is a
"publish" block that can be added to the catalog.bom.)

---
With Svet's suggestion, if there was no manifest section/file, then
should we auto-generate something from the item(s) in the catalog.bom?

I can see how that could easily work for a .bom file that has a single
item (e.g. we use the catalog item's id + version, possibly with a
special prefix in the symbolic name to avoid conflicts).

However, if there are multiple items then it would get trickier.

I'm inclined to say that for a minimal viable product (MVP) we can
insist on the "manifest" section in the catalog.bom.

Aled


On 20/12/2016 16:34, Svetoslav Neykov wrote:
>> Svet, if instead we tried to infer it from the catalog.bom, would we
require some additional metadata within the .bom file? Or would we use the
catalog item's id + version? I'm not convinced by the latter - it would
mean some .bom files would work and others wouldn't (e.g. if the .bom had
multiple items with different versions). Better to support the explicit
approach IMO.
> I imagine it would be additional metadata. On the other hand I don't see
a technical reason why we need an explicit symbolicName and version - they
can be auto-generated.
>
> Svet.
>
>
>> On 20.12.2016 г., at 17:50, Aled Sage <al...@gmail.com> wrote:
>>
>> Hi all,
>>
>> +1
>>
>> (D) sounds good. What version are you imagining the bundle would be, if
one runs `br catalog add ~/my/project/ --name com.example.myproject`?
>>
>> ---
>> I like the idea of uploading a plain zip (rather than only supporting
OSGi bundles) - that makes it simpler for non-java folk. The use of OSGi
becomes a (hidden) implementation detail to many users.
>>
>> ---
>> If auto-generating the manifest, I think we need the user to be explicit
about symbolic name and version. Having these supplied in the REST api call
(as Alex suggests) would achieve that.
>>
>> Svet, if instead we tried to infer it from the catalog.bom, would we
require some additional metadata within the .bom file? Or would we use the
catalog item's id + version? I'm not convinced by the latter - it would
mean some .bom files would work and others wouldn't (e.g. if the .bom had
multiple items with different versions). Better to support the explicit
approach IMO.
>>
>> ---
>> For E ("have a mechanism whereby deployed entities based on an affected
blueprint are optionally migrated to the new code"), that feels like a
separate discussion. It could equally apply to a pure YAML .bom file that
has been added to the catalog.
>>
>> I suggest we discuss that in a separate email thread.
>>
>> ---
>> For (G), it's an interesting suggestion from Svet to make use of Karaf
Cellar for HA nodes. I'm hesitant (e.g. if restarting a standalone Brooklyn
node whose VM has died, then it adds big additional requirements for what
constitutes the "persisted state"). On the other hand, it's good to use
well-established technologies rather than re-inventing things!
>>
>> An alternative ("pure brooklyn") approach could be to write the bundle
to persisted state; on rebind, we'd install + activate those bundles.
>>
>> ---
>> For "catalogGroupId", I agree with Svet that in the initial use-case
this can be an implementation detail.
>>
>> It could be set as the bundle's symbolic name + version: everything from
the bundle should be deleted at once, along with the bundle.
>>
>> Longer term, I can see how exposing "catalogGroupId" to the user could
support more use-cases (e.g. for several catalog items from different
bundles to work together). I don't think we should try to support that yet.
>>
>> Aled
>>
>>
>> On 19/12/2016 17:19, Geoff Macartney wrote:
>>> hi Alex,
>>>
>>> this looks like a good feature to have, I shall look at the PR as soon
as I
>>> can.
>>>
>>> The catalog.bom scanner feature was initially enabled by default, but we
>>> had to
>>> disable it because it turned out not to work properly with rebind. I
don't
>>> think
>>> it should be a lot of work to fix that but it hasn't been something
we've
>>> got round
>>> to yet.  This would be a great opportunity to look back at that.
>>>
>>> Some random thoughts:
>>>
>>> re (C), if we are going to treat the zips as bundles, my gut feel is
that
>>> we
>>> should insist on a manifest and get the metadata from it.  It doesn't
feel
>>> to me
>>> like it makes much sense to allow a zip file without a MANIFEST.MF but
>>> convey
>>> the intended bundle metadata to Brooklyn via HTTP headers.  And rather
than
>>> infer bundle metadata I think it's better to ask users to be explicit
about
>>> what
>>> their intentions are.  To make users lives easier, we could
>>> add a command to br to generate the manifest (locally) with correct
syntax,
>>> so that the manifest is in the right place, rather than have br add the
data
>>> to the "upload" request headers.
>>>
>>> re. (D) will be glad to have a look at it
>>>
>>> re. (E) it would certainly need to be optional - maybe keep it as an
>>> explicit
>>> separate command ('upgrade'?)
>>>
>>> (F) it does seem like a lot of work but might be nice for users who are
not
>>> keen on command lines.
>>>
>>> G - I:  we'll definitely need to pay close attention to persistence and
>>> rebind;
>>> I wonder also about HA operation, are there any additional implications?
>>>
>>> (J) I think it would be good to treat all the files from a jar, sorry
>>> bundle,
>>> as an atomic group - cleaner that way perhaps than allowing
delete/update
>>> of
>>> individual entries from a bundle on a piecemeal basis.  Rest support on
>>> delete
>>> catalog could warn about related catalog entries being deleted and ask
for
>>> a "--force" param to confirm.
>>>
>>> Geoff
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Fri, 16 Dec 2016 at 15:24 Svetoslav Neykov <
>>> svetoslav.neykov@cloudsoftcorp.com> wrote:
>>>
>>>> +1
>>>>
>>>> Some thoughts:
>>>>    * (A) add a utility class BundleMaker
>>>>      Sounds very similar to
>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles <
>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles>
>>>>      Looking at the code it's much more focused on zip files so I guess
>>>> there's no much overlap, but worth keeping in mind
>>>>    * (C) accept bundle symbolic name and version
>>>>      Why require them at all? Could infer them from the catalog.bom in
some
>>>> way - maybe require those properties to be in there. If not present are
>>>> they really needed?
>>>>    * (G) Bundles installed via this mechanism are not persisted
currently &
>>>> (I) We persist the individual catalog items as YAML, so we end up with
two
>>>> records
>>>>      Suggest marking the catalog items coming from bundles as
>>>> non-persistable. Then try to share the bundles between HA nodes. (Karaf
>>>> Cellar?)
>>>>    * (J) Introduce a catalogGroupId field on catalog items;
>>>>      Agree this could be useful and I like the idea of deleting the
bundle
>>>> altogether with the catalog items. From user's perspective I don't see
the
>>>> need for an extra field (i.e. it's an implementation detail).
>>>>
>>>> Svet.
>>>>
>>>>
>>>>> On 16.12.2016 г., at 12:50, Alex Heneveld <
>>>> alex.heneveld@cloudsoftcorp.com> wrote:
>>>>> Hi Brooklyners-
>>>>>
>>>>> In the code we currently have two routes for users to install new
>>>>> blueprints:
>>>>>
>>>>> (1) upload a catalog YAML file to /v1/catalog
>>>>>
>>>>> (2) install a bundle with catalog.bom in the root
>>>>>
>>>>> The feature (2) is disabled by default, but I'd like to move towards
>>>>> enabling it.  This will make it easier to create nicely structured BOM
>>>>> files because scripts etc can be taken out of the BOM, stored as
files in
>>>>> the same bundle.  (Because URLs of the form
>>>>> `classpath://scripts/install.sh`  use the bundle's classpath to
resolve.)
>>>>>
>>>>> As a first step in #485 [1] I do a few things:
>>>>>
>>>>> (A) add a utility class BundleMaker that lets us create and modify
>>>>> bundles/zips, to make it easier to do things we might want to with
>>>> bundles,
>>>>> especially for testing
>>>>>
>>>>> (B) add an endpoint to the REST API which allows uploading a bundle
ZIP
>>>>>
>>>>> (C) accept bundle symbolic name and version in that REST API to
>>>> facilitate
>>>>> uploading non-bundle ZIPs where the OSGi MANIFEST.MF is automatically
>>>>> generated
>>>>>
>>>>> With this PR, if you have a directory on your local file system with
>>>>> scripts and config files, and a BOM which refers to them, you can just
>>>> ZIP
>>>>> that up an upload it, specifying the bundle name so that a YAML
blueprint
>>>>> author never needs to touch any java-isms.
>>>>>
>>>>> Where I see this going is a development workflow where a user can edit
>>>>> files locally and upload the ZIP to have that installed, and if they
make
>>>>> changes locally they can POST it again to have catalog items updated
>>>>> (because default version is a SNAPSHOT).  We could also:
>>>>>
>>>>> (D) have `br catalog add ~/my/project/ --name my.project` create a ZIP
>>>> and
>>>>> POST it, with bundle name metadata, so essentially the user's process
is
>>>>> just to run that whenever they make a change
>>>>>
>>>>> (E) have a mechanism whereby deployed entities based on an affected
>>>>> blueprint are optionally migrated to the new code, so if you've
changed
>>>> an
>>>>> enricher the changes are picked up, or if say a launch.sh script has
>>>>> changed, a restart will run the new code
>>>>>
>>>>> The above are fairly straightforward programmatically (although good
user
>>>>> interaction with (E) needs some thought).  So I think we can pretty
>>>> quickly
>>>>> get to a much smoother dev workflow.
>>>>>
>>>>>
>>>>> That's the highlight of this message.  You can jump to the end, unless
>>>>> you're interested in some important but low level details...
>>>>>
>>>>>
>>>>> I'm also tempted by:
>>>>>
>>>>> (F) Integration with web-based IDE and/or Brooklyn reading and writing
>>>>> straight from GitHub -- but this seems like a lot of work and I'm not
>>>>> convinced it's much better than (D) workflow-wise
>>>>>
>>>>> Before we can change (2) to be the default, or start widely using the
>>>> POST
>>>>> a ZIP feature, we need to sort out some issues to do with persistence
and
>>>>> reloading:
>>>>>
>>>>> (G) Bundles installed via this mechanism are not persisted currently,
so
>>>> if
>>>>> you move to a different Brooklyn using the same backing store, you'll
>>>> lose
>>>>> those bundles
>>>>>
>>>>> (H) On rebind, bundles aren't always activated when needed, meaning
items
>>>>> can't be loaded
>>>>>
>>>>> (I) We persist the individual catalog items as YAML, so we end up with
>>>> two
>>>>> records — the YAML from the catalog.bom in the bundle, and the YAML
>>>>> persisted for the item.  This isn't a problem per se, but something to
>>>>> think about, and some sometimes surprising behaviour.  In particular
if
>>>> you
>>>>> delete the persisted YAML, the bundle is still there so the item is no
>>>>> longer deleted after a full rebind.
>>>>>
>>>>> One idea which might be useful is:
>>>>>
>>>>> (J) Introduce a catalogGroupId field on catalog items; this will do
two
>>>>> things:  if you try to delete an item with such a record, you'll be
>>>>> encouraged to delete all such items (maybe disallowed to delete an
>>>>> individual one), with the effect of deleting the bundle if it comes
from
>>>> a
>>>>> bundle; and when resolving types we search first for items with the
same
>>>>> catalogGroupId (so that e.g. if I install MyCluster:1.0 and
MyNode:1.0 in
>>>>> the same group, the former can refer simply to "MyNode" but if I
install
>>>> a
>>>>> 2.0 version of that group, the 1.0 cluster still loads the 1.0 node --
>>>> this
>>>>> has bitten people i the past)
>>>>>
>>>>> There is a related Brooklyn upgrade problem worth mentioning, which
the
>>>>> above might help with, where:
>>>>>
>>>>> (K) If I migrate from Brooklyn 10 to 11 when it comes out, I'll no
longer
>>>>> have certain entities that were at v10, since we don't include those;
an
>>>>> upgrade could include rules that certain groupIds need to be updated,
or
>>>> it
>>>>> can search and attempt to automatically apply the updates
>>>>>
>>>>>
>>>>> Quite a lot here and we don't need to solve it but I wanted to:
>>>>>
>>>>> * Share the current thinking
>>>>>
>>>>> * Get opinions on the general dev workflow suggested by (D)
>>>>>
>>>>>
>>>>> Thanks for feedback -- and if we like it help with (D) would be
>>>> appreciated!
>>>>> Best
>>>>> Alex
>>>>>
>>>>>
>>>>>
>>>>> [1] . https://github.com/apache/brooklyn-server/pull/485