Externalised configuration in brooklyn.properties

[Richard Downer <ri...@apache.org>]

All,

Alasdair Hodge recently contributed support for "externalised
configuration", where a blueprint can indicate that a value is to be
loaded from an external store rather than being coded directly. This
is particularly useful to prevent plain-text passwords being placed in
blueprints, but it has many uses.

It's not valid outside of blueprints however. I would like to be able
to use it on configuration stored in brooklyn.properties. I've been
investigating options for doing this and I would like to share what I
have discovered and see what the consensus is for how to implement
this.

*Where to implement it*

brooklyn.properties is loaded by the BrooklynProperties class
(surprisingly). On first inspection this would be a good place to
resolve the configuration; however resolving the configuration
requires a reference to a management context, which this class doesn't
have.

In fact resolving the configuration at this stage is actually
impossible, because the configuration resolvers are configured in
brooklyn.properties! Therefore you must have loaded
brooklyn.properties, and configured a management context, before you
can resolve the configuration.

My next attempt was to implement it in the constructor of
AbstractManagementContext - at the end of the constructor (crucially,
after brooklyn.properties has been loaded, and the external config
supplier registry has been set), the configMap is transformed and all
config resolved. This does appear to be usable.

*How to refer to external config*

In the YAML blueprints this is relatively easy, as there is already a
framework for parsing the YAML and a DSL that can be extended to add
the new configuration lookup. For brooklyn.properties there is no such
thing - every property value is a string literal.

Ideally we'd use the same DSL as the YAML parser, but this could be a
complex solution, since the DSL parser is embedded into the deployment
plan parser. We could write a new parser, but that seems to be
over-complicated.

For the moment I have taken a very simple approach - if the property
value matches *in its entirety*
"$brooklyn:external:supplierName:keyName" then an external config key
will be made.

This seems to be working. It's familiar to the DSL user, but it's also
different enough that it could cause confusion. It's also not really
flexible enough - it can replace the *whole* value with something
looked up from storage, but it's not possible to mix plain text with
multiple lookups, as could be done in YAML.

What do people think?

Thanks
Richard.

Re: Externalised configuration in brooklyn.properties

[Richard Downer <ri...@apache.org>]

Ignore me. Entity getConfig() doesn't inherit brooklyn.properties
settings so that's an irrelevant place to implement it.

Richard.


On 9 November 2015 at 15:40, Richard Downer <ri...@apache.org> wrote:
> I haven't yet looked at the DSL resolver, but I have identified a
> possible issue with this approach.
>
> The external config supplier spec dictates that the config supplier is
> checked every time the configuration is referenced. However, the
> approach I have used will resolve the configuration exactly once,
> during management context initialisation.
>
> Are there any other places where I could "hook in" to do this? I'm
> looking at parts related to Entity, but that has its own problems:
> unless done in exactly the right place, it still wouldn't match the
> behaviour that configuration needs to be resolved when the blueprint
> is instantiated. It could end up in the opposite extreme e.g.
> configuration is resolved every time the entity does getConfig(),
> which breaks our guarantee that entity configuration doesn't change
> during runtime except in special conditions.
>
> Thanks
> Richard.
>
> On 4 November 2015 at 17:40, Alex Heneveld
> <al...@cloudsoftcorp.com> wrote:
>> Good ideas Richard. This workaround feels good enough to me for the short
>> term.
>>
>> I agree the ideal would be to tie in to the dsl resolver - which actually
>> should be pretty easy to disentangle (see mail thread w Martin from a month
>> ago) - when props are accessed. That might be worth a few hours' spike.
>>
>> Best
>> Alex
>> On 4 Nov 2015 07:59, "Richard Downer" <ri...@apache.org> wrote:
>>
>>> All,
>>>
>>> Alasdair Hodge recently contributed support for "externalised
>>> configuration", where a blueprint can indicate that a value is to be
>>> loaded from an external store rather than being coded directly. This
>>> is particularly useful to prevent plain-text passwords being placed in
>>> blueprints, but it has many uses.
>>>
>>> It's not valid outside of blueprints however. I would like to be able
>>> to use it on configuration stored in brooklyn.properties. I've been
>>> investigating options for doing this and I would like to share what I
>>> have discovered and see what the consensus is for how to implement
>>> this.
>>>
>>> *Where to implement it*
>>>
>>> brooklyn.properties is loaded by the BrooklynProperties class
>>> (surprisingly). On first inspection this would be a good place to
>>> resolve the configuration; however resolving the configuration
>>> requires a reference to a management context, which this class doesn't
>>> have.
>>>
>>> In fact resolving the configuration at this stage is actually
>>> impossible, because the configuration resolvers are configured in
>>> brooklyn.properties! Therefore you must have loaded
>>> brooklyn.properties, and configured a management context, before you
>>> can resolve the configuration.
>>>
>>> My next attempt was to implement it in the constructor of
>>> AbstractManagementContext - at the end of the constructor (crucially,
>>> after brooklyn.properties has been loaded, and the external config
>>> supplier registry has been set), the configMap is transformed and all
>>> config resolved. This does appear to be usable.
>>>
>>> *How to refer to external config*
>>>
>>> In the YAML blueprints this is relatively easy, as there is already a
>>> framework for parsing the YAML and a DSL that can be extended to add
>>> the new configuration lookup. For brooklyn.properties there is no such
>>> thing - every property value is a string literal.
>>>
>>> Ideally we'd use the same DSL as the YAML parser, but this could be a
>>> complex solution, since the DSL parser is embedded into the deployment
>>> plan parser. We could write a new parser, but that seems to be
>>> over-complicated.
>>>
>>> For the moment I have taken a very simple approach - if the property
>>> value matches *in its entirety*
>>> "$brooklyn:external:supplierName:keyName" then an external config key
>>> will be made.
>>>
>>> This seems to be working. It's familiar to the DSL user, but it's also
>>> different enough that it could cause confusion. It's also not really
>>> flexible enough - it can replace the *whole* value with something
>>> looked up from storage, but it's not possible to mix plain text with
>>> multiple lookups, as could be done in YAML.
>>>
>>> What do people think?
>>>
>>> Thanks
>>> Richard.
>>>

Re: Externalised configuration in brooklyn.properties

[Richard Downer <ri...@apache.org>]

I haven't yet looked at the DSL resolver, but I have identified a
possible issue with this approach.

The external config supplier spec dictates that the config supplier is
checked every time the configuration is referenced. However, the
approach I have used will resolve the configuration exactly once,
during management context initialisation.

Are there any other places where I could "hook in" to do this? I'm
looking at parts related to Entity, but that has its own problems:
unless done in exactly the right place, it still wouldn't match the
behaviour that configuration needs to be resolved when the blueprint
is instantiated. It could end up in the opposite extreme e.g.
configuration is resolved every time the entity does getConfig(),
which breaks our guarantee that entity configuration doesn't change
during runtime except in special conditions.

Thanks
Richard.

On 4 November 2015 at 17:40, Alex Heneveld
<al...@cloudsoftcorp.com> wrote:
> Good ideas Richard. This workaround feels good enough to me for the short
> term.
>
> I agree the ideal would be to tie in to the dsl resolver - which actually
> should be pretty easy to disentangle (see mail thread w Martin from a month
> ago) - when props are accessed. That might be worth a few hours' spike.
>
> Best
> Alex
> On 4 Nov 2015 07:59, "Richard Downer" <ri...@apache.org> wrote:
>
>> All,
>>
>> Alasdair Hodge recently contributed support for "externalised
>> configuration", where a blueprint can indicate that a value is to be
>> loaded from an external store rather than being coded directly. This
>> is particularly useful to prevent plain-text passwords being placed in
>> blueprints, but it has many uses.
>>
>> It's not valid outside of blueprints however. I would like to be able
>> to use it on configuration stored in brooklyn.properties. I've been
>> investigating options for doing this and I would like to share what I
>> have discovered and see what the consensus is for how to implement
>> this.
>>
>> *Where to implement it*
>>
>> brooklyn.properties is loaded by the BrooklynProperties class
>> (surprisingly). On first inspection this would be a good place to
>> resolve the configuration; however resolving the configuration
>> requires a reference to a management context, which this class doesn't
>> have.
>>
>> In fact resolving the configuration at this stage is actually
>> impossible, because the configuration resolvers are configured in
>> brooklyn.properties! Therefore you must have loaded
>> brooklyn.properties, and configured a management context, before you
>> can resolve the configuration.
>>
>> My next attempt was to implement it in the constructor of
>> AbstractManagementContext - at the end of the constructor (crucially,
>> after brooklyn.properties has been loaded, and the external config
>> supplier registry has been set), the configMap is transformed and all
>> config resolved. This does appear to be usable.
>>
>> *How to refer to external config*
>>
>> In the YAML blueprints this is relatively easy, as there is already a
>> framework for parsing the YAML and a DSL that can be extended to add
>> the new configuration lookup. For brooklyn.properties there is no such
>> thing - every property value is a string literal.
>>
>> Ideally we'd use the same DSL as the YAML parser, but this could be a
>> complex solution, since the DSL parser is embedded into the deployment
>> plan parser. We could write a new parser, but that seems to be
>> over-complicated.
>>
>> For the moment I have taken a very simple approach - if the property
>> value matches *in its entirety*
>> "$brooklyn:external:supplierName:keyName" then an external config key
>> will be made.
>>
>> This seems to be working. It's familiar to the DSL user, but it's also
>> different enough that it could cause confusion. It's also not really
>> flexible enough - it can replace the *whole* value with something
>> looked up from storage, but it's not possible to mix plain text with
>> multiple lookups, as could be done in YAML.
>>
>> What do people think?
>>
>> Thanks
>> Richard.
>>

Re: Externalised configuration in brooklyn.properties

[Alex Heneveld <al...@cloudsoftcorp.com>]

Good ideas Richard. This workaround feels good enough to me for the short
term.

I agree the ideal would be to tie in to the dsl resolver - which actually
should be pretty easy to disentangle (see mail thread w Martin from a month
ago) - when props are accessed. That might be worth a few hours' spike.

Best
Alex
On 4 Nov 2015 07:59, "Richard Downer" <ri...@apache.org> wrote:

> All,
>
> Alasdair Hodge recently contributed support for "externalised
> configuration", where a blueprint can indicate that a value is to be
> loaded from an external store rather than being coded directly. This
> is particularly useful to prevent plain-text passwords being placed in
> blueprints, but it has many uses.
>
> It's not valid outside of blueprints however. I would like to be able
> to use it on configuration stored in brooklyn.properties. I've been
> investigating options for doing this and I would like to share what I
> have discovered and see what the consensus is for how to implement
> this.
>
> *Where to implement it*
>
> brooklyn.properties is loaded by the BrooklynProperties class
> (surprisingly). On first inspection this would be a good place to
> resolve the configuration; however resolving the configuration
> requires a reference to a management context, which this class doesn't
> have.
>
> In fact resolving the configuration at this stage is actually
> impossible, because the configuration resolvers are configured in
> brooklyn.properties! Therefore you must have loaded
> brooklyn.properties, and configured a management context, before you
> can resolve the configuration.
>
> My next attempt was to implement it in the constructor of
> AbstractManagementContext - at the end of the constructor (crucially,
> after brooklyn.properties has been loaded, and the external config
> supplier registry has been set), the configMap is transformed and all
> config resolved. This does appear to be usable.
>
> *How to refer to external config*
>
> In the YAML blueprints this is relatively easy, as there is already a
> framework for parsing the YAML and a DSL that can be extended to add
> the new configuration lookup. For brooklyn.properties there is no such
> thing - every property value is a string literal.
>
> Ideally we'd use the same DSL as the YAML parser, but this could be a
> complex solution, since the DSL parser is embedded into the deployment
> plan parser. We could write a new parser, but that seems to be
> over-complicated.
>
> For the moment I have taken a very simple approach - if the property
> value matches *in its entirety*
> "$brooklyn:external:supplierName:keyName" then an external config key
> will be made.
>
> This seems to be working. It's familiar to the DSL user, but it's also
> different enough that it could cause confusion. It's also not really
> flexible enough - it can replace the *whole* value with something
> looked up from storage, but it's not possible to mix plain text with
> multiple lookups, as could be done in YAML.
>
> What do people think?
>
> Thanks
> Richard.
>