Multiple formats for catalog add and for deploy

[Alex Heneveld <al...@cloudsoft.io>]

Hi Brooklyners,

Since the Catalog was overhauled many years ago with the TypeRegistry 
approach, we have had support in the back-end for custom types and for 
custom plan formats.  The former is useful e.g. if you want to add a new 
type to be used as a config key or initializer.  The latter is useful if 
the usual entity-centric CAMP YAML isn't the native format.  The latter 
is extensible, using the TypePlanTransformer OSGi service at runtime, so 
new bundles can register services to support new format types -- e.g. 
TOSCA, Kubernetes, pick your poison.  The only requirements are that the 
service register the type saying what its supertypes are and is then 
able to instantiate it when requested (using whatever it wants, 
including calling to a CLI or REST endpoint).  We've also added a 
BeanWithType transformer (specify "format: bean-with-type" in the 
catalog BOM) to use simple POJO/Jackson YAML deserialization, which has 
been really useful for Initializers.

There are two related features that would complete this nicely:

* Offer a similar extensible approach to resolving the overall 
artifact/bundle added to the catalog:  currently _within_ the 
catalog.bom file one can define the format for items, but the BOM YAML 
format is required, and there is a simple try/catch to allow upload of 
just a BOM YAML or a ZIP which contains either catalog.bom or OSGi 
metadata (or both); having an extensible "services" mechanism for 
scoring and resolving catalog bundles would allow Apache Brooklyn to 
handle other items being added to the catalog in their native format 
with no extra metadata. Similar to TypePlanTransformer, the e.g. 
BundleInstallResolver would be responsible for scoring its applicability 
and adding the items to the catalog.  It could lean on the existing 
catalog.bom format to do so, or it could replace that with its own 
reading of what items are within the bundle.  For instance a 
BundleInstallResolver could accept a Dockerfile or a Helm Chart or a 
TOSCA CSAR or a CloudFoundry project, and make it known within the catalog.

* On the REST API, when deploying or adding to catalog, add an argument 
so that the caller can specify the format of the artifact being deployed 
or added to the catalog.  This will link one-to-one with:  for 
deploying, the TypePlanTransformer; and for adding to catalog, the (new) 
BundleInstallResolver.  By default auto-detect will operate, asking 
services to score their applicability.  (eg. if it's a ZIP with a 
catalog.bom the current OsgiArchiveInstaller will score 1.0, if it's 
OSGi but no catalog.bom then score 0.2.) Currently, deployment will use 
auto-detect against all TypePlanTransformers, with no way on the API to 
specify a format. And for adding to catalog, it is only the BOM YAML or 
ZIP-with-BOM-and-or-OSGi formats that are supported.

I think this could go a long way to making Apache Brooklyn more 
accessible, because with just a small code investment, the software can 
then support potentially many other existing deployment formats, 
reducing (removing!) the need for a consumer to learn the Apache 
Brooklyn syntax for plans and bundles.

I propose to work on the above and welcome any thoughts from the community.

Once the API is updated it would be good to add a CLI argument to 
support this.  I might could do with help here!

Best
Alex

Re: Multiple formats for catalog add and for deploy

[Alex Heneveld <al...@cloudsoft.io>]

Hi Geoff-

That's exactly the type of thing I was thinking.

I'd probably go with an "format" API parameter rather than a header 
since it isn't a standard MIME type but that's a detail.

For deployments you are correct, the registered transformer for the 
service would have to be able to tell where it should be deployed.  Not 
many systems have a multi-location concept like Apache Brooklyn does so 
you'd have to "cheat" in many cases.  But any of the following cheats 
seem okay to me:

* the transformer has a default location that it expects to be set up, 
e.g. one called "docker.default.location" registered in Brooklyn, and 
uses it
* the location is specified by extension metadata in the plan or even a 
comment, e.g. the Dockerfile has a line "# brooklyn-deploy: 
my-docker-target"
* you wrap the foreign type, so we register a type whose implementation 
is a dockerfile, as "my-docker-file", then deploy
     "{ services: [ { type: my-docker-file } ], location: 
my-docker-target }"

While the last one looks pointless -- and on its own it is -- it allows 
you to compose multiple things together, eg docker, k8s, terraform, 
tosca, cloudformation, arm, cloud foundry, etc, and add Apache Brooklyn 
policies to it.  As people combine cloud services with container 
platforms within the same service, and as they want consistent 
management across them, I think that becomes a strong suit for Apache 
Brooklyn.

For k8s I could see "br catalog add ." running over a directory 
containing a helm chart, and a bundle gets added with the deployments 
from the chart, which you could then deploy (or compose) using standard 
Brooklyn CAMP syntax.  Or simpler, "br deploy --file 
myk8s.deployment.yaml" if Brooklyn has been configured with a 
"kubernetes.default.location" pointing at your KubernetesLocation (and 
the k8s transformer written so that that is what it uses).

PS - I expect a first draft of the PR to land shortly

HTH
Alex



On 07/10/2020 21:41, Geoff Macartney wrote:
> Hi Alex,
>
> That certainly sounds like an interesting idea. I'm not entirely sure
> I've got my head round it though. I guess I can see how you could add
> a new type to the catalog, say by posting a Dockerfile as the request
> body and having an API (HTTP) header of `x-brooklyn-type: dockerfile`.
> In this case I expect the value "dockerfile" is some format specifier
> registered with your BundleInstallResolver, e.g. by a bundle that
> knows how to deploy to Docker.
>
> I'm a little less clear how that could work for deployments. You can't
> just post a Dockerfile to deploy, you'd have to at least add a bit of
> Brooklyn meta-data, namely a location (`location: my-local-docker`)
> and application name.
>
> Could you maybe sketch out what the requests would look like to add a
> Dockerfile to the catalog, and to deploy it to a Docker instance?
>
> What about, say, a K8s deployment descriptor, again to add it to
> catalog, and to deploy it to a KubernetesLocation?
>
> Cheers
> Geoff
>
>
> On Mon, 5 Oct 2020 at 13:41, Alex Heneveld <al...@cloudsoft.io> wrote:
>>
>> Hi Brooklyners,
>>
>> Since the Catalog was overhauled many years ago with the TypeRegistry
>> approach, we have had support in the back-end for custom types and for
>> custom plan formats.  The former is useful e.g. if you want to add a new
>> type to be used as a config key or initializer.  The latter is useful if
>> the usual entity-centric CAMP YAML isn't the native format.  The latter
>> is extensible, using the TypePlanTransformer OSGi service at runtime, so
>> new bundles can register services to support new format types -- e.g.
>> TOSCA, Kubernetes, pick your poison.  The only requirements are that the
>> service register the type saying what its supertypes are and is then
>> able to instantiate it when requested (using whatever it wants,
>> including calling to a CLI or REST endpoint).  We've also added a
>> BeanWithType transformer (specify "format: bean-with-type" in the
>> catalog BOM) to use simple POJO/Jackson YAML deserialization, which has
>> been really useful for Initializers.
>>
>> There are two related features that would complete this nicely:
>>
>> * Offer a similar extensible approach to resolving the overall
>> artifact/bundle added to the catalog:  currently _within_ the
>> catalog.bom file one can define the format for items, but the BOM YAML
>> format is required, and there is a simple try/catch to allow upload of
>> just a BOM YAML or a ZIP which contains either catalog.bom or OSGi
>> metadata (or both); having an extensible "services" mechanism for
>> scoring and resolving catalog bundles would allow Apache Brooklyn to
>> handle other items being added to the catalog in their native format
>> with no extra metadata. Similar to TypePlanTransformer, the e.g.
>> BundleInstallResolver would be responsible for scoring its applicability
>> and adding the items to the catalog.  It could lean on the existing
>> catalog.bom format to do so, or it could replace that with its own
>> reading of what items are within the bundle.  For instance a
>> BundleInstallResolver could accept a Dockerfile or a Helm Chart or a
>> TOSCA CSAR or a CloudFoundry project, and make it known within the catalog.
>>
>> * On the REST API, when deploying or adding to catalog, add an argument
>> so that the caller can specify the format of the artifact being deployed
>> or added to the catalog.  This will link one-to-one with:  for
>> deploying, the TypePlanTransformer; and for adding to catalog, the (new)
>> BundleInstallResolver.  By default auto-detect will operate, asking
>> services to score their applicability.  (eg. if it's a ZIP with a
>> catalog.bom the current OsgiArchiveInstaller will score 1.0, if it's
>> OSGi but no catalog.bom then score 0.2.) Currently, deployment will use
>> auto-detect against all TypePlanTransformers, with no way on the API to
>> specify a format. And for adding to catalog, it is only the BOM YAML or
>> ZIP-with-BOM-and-or-OSGi formats that are supported.
>>
>> I think this could go a long way to making Apache Brooklyn more
>> accessible, because with just a small code investment, the software can
>> then support potentially many other existing deployment formats,
>> reducing (removing!) the need for a consumer to learn the Apache
>> Brooklyn syntax for plans and bundles.
>>
>> I propose to work on the above and welcome any thoughts from the community.
>>
>> Once the API is updated it would be good to add a CLI argument to
>> support this.  I might could do with help here!
>>
>> Best
>> Alex
>>
>>

Re: Multiple formats for catalog add and for deploy

[Geoff Macartney <ge...@gmail.com>]

Hi Alex,

That certainly sounds like an interesting idea. I'm not entirely sure
I've got my head round it though. I guess I can see how you could add
a new type to the catalog, say by posting a Dockerfile as the request
body and having an API (HTTP) header of `x-brooklyn-type: dockerfile`.
In this case I expect the value "dockerfile" is some format specifier
registered with your BundleInstallResolver, e.g. by a bundle that
knows how to deploy to Docker.

I'm a little less clear how that could work for deployments. You can't
just post a Dockerfile to deploy, you'd have to at least add a bit of
Brooklyn meta-data, namely a location (`location: my-local-docker`)
and application name.

Could you maybe sketch out what the requests would look like to add a
Dockerfile to the catalog, and to deploy it to a Docker instance?

What about, say, a K8s deployment descriptor, again to add it to
catalog, and to deploy it to a KubernetesLocation?

Cheers
Geoff


On Mon, 5 Oct 2020 at 13:41, Alex Heneveld <al...@cloudsoft.io> wrote:
>
>
> Hi Brooklyners,
>
> Since the Catalog was overhauled many years ago with the TypeRegistry
> approach, we have had support in the back-end for custom types and for
> custom plan formats.  The former is useful e.g. if you want to add a new
> type to be used as a config key or initializer.  The latter is useful if
> the usual entity-centric CAMP YAML isn't the native format.  The latter
> is extensible, using the TypePlanTransformer OSGi service at runtime, so
> new bundles can register services to support new format types -- e.g.
> TOSCA, Kubernetes, pick your poison.  The only requirements are that the
> service register the type saying what its supertypes are and is then
> able to instantiate it when requested (using whatever it wants,
> including calling to a CLI or REST endpoint).  We've also added a
> BeanWithType transformer (specify "format: bean-with-type" in the
> catalog BOM) to use simple POJO/Jackson YAML deserialization, which has
> been really useful for Initializers.
>
> There are two related features that would complete this nicely:
>
> * Offer a similar extensible approach to resolving the overall
> artifact/bundle added to the catalog:  currently _within_ the
> catalog.bom file one can define the format for items, but the BOM YAML
> format is required, and there is a simple try/catch to allow upload of
> just a BOM YAML or a ZIP which contains either catalog.bom or OSGi
> metadata (or both); having an extensible "services" mechanism for
> scoring and resolving catalog bundles would allow Apache Brooklyn to
> handle other items being added to the catalog in their native format
> with no extra metadata. Similar to TypePlanTransformer, the e.g.
> BundleInstallResolver would be responsible for scoring its applicability
> and adding the items to the catalog.  It could lean on the existing
> catalog.bom format to do so, or it could replace that with its own
> reading of what items are within the bundle.  For instance a
> BundleInstallResolver could accept a Dockerfile or a Helm Chart or a
> TOSCA CSAR or a CloudFoundry project, and make it known within the catalog.
>
> * On the REST API, when deploying or adding to catalog, add an argument
> so that the caller can specify the format of the artifact being deployed
> or added to the catalog.  This will link one-to-one with:  for
> deploying, the TypePlanTransformer; and for adding to catalog, the (new)
> BundleInstallResolver.  By default auto-detect will operate, asking
> services to score their applicability.  (eg. if it's a ZIP with a
> catalog.bom the current OsgiArchiveInstaller will score 1.0, if it's
> OSGi but no catalog.bom then score 0.2.) Currently, deployment will use
> auto-detect against all TypePlanTransformers, with no way on the API to
> specify a format. And for adding to catalog, it is only the BOM YAML or
> ZIP-with-BOM-and-or-OSGi formats that are supported.
>
> I think this could go a long way to making Apache Brooklyn more
> accessible, because with just a small code investment, the software can
> then support potentially many other existing deployment formats,
> reducing (removing!) the need for a consumer to learn the Apache
> Brooklyn syntax for plans and bundles.
>
> I propose to work on the above and welcome any thoughts from the community.
>
> Once the API is updated it would be good to add a CLI argument to
> support this.  I might could do with help here!
>
> Best
> Alex
>
>