You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@polygene.apache.org by Niclas Hedhman <ni...@hedhman.org> on 2017/05/29 05:47:52 UTC
Configuration structure
Hi,
at the moment, the content of the properties file drives what is expected
in the ConfigurationComposite. If there are more properties than there are
declared and matching Proprety<> methods, then there is an Exception.
This might have been rational back in the days when this was discussed
first time, but if we are heading towards supporting external and perhaps
more exotic configuration "supply-chains", then I think it would be more
logical that the ConfigurationComposite simply reads what it wants and
ignores everything else.
AND then would could have this super cool addition that if no files are
found that works, fall back to System.getProperties() and System.getenv() as
the final backups.
WDYT?
Cheers
--
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java
Re: Configuration structure
Posted by Niclas Hedhman <ni...@hedhman.org>.
Yes, to everything;
It is fallback if properties file, json, xml or yaml files can't be found,
then it will do this, and yes, if the config es is persisting it will not
be using them next start.
And yes, both System properties and Environment variables are "copies" of
the System.getProperties() and System.getenv() respectively. SO, should be
no problem that those sources are mutable any more than properties files
are also mutable.
Cheers
On Sun, Jun 4, 2017 at 6:39 PM, Paul Merlin <pa...@apache.org> wrote:
> Le 2017-06-04 12:21, Niclas Hedhman a écrit :
>
>> Well, I implemented the simple/hacky way now.
>>
>> public interface MyConfig
>> {
>> Property<String> osName();
>>
>> Property<String> home();
>>
>> Property<String> path();
>> }
>>
>>
>> will now read "os.name" from System Properties and $HOME and $PATH/%PATH%
>> of Environment variables.
>>
>> Additionally, System properties takes precedence over environment
>> variables, so env variables can be overridden more easily.
>>
>
> This is to get defaults if the configuration could not be found in the
> corresponding ES, correct?
>
> So, the values gathered from system properties and environment variables
> during that "first run" will then be persisted into the corresponding ES.
> If the ES is persistent, then the next run of the application will not use
> system properties nor environment variables but the values fetched from the
> configuration ES, that is values captured from "the environment" during the
> first run.
>
> Of course, if using a non-persistent ES for configuration entities (e.g.
> MemoryES), then "the environment" will be used on each run.
>
> I didn't think this through but it could get surprising.
>
> As a side note, system properties and environment variable are mutable. So
> this new mechanism is in fact taking a copy of their values at some point
> in time, application startup, or later service activation.
>
>
>
--
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java
Re: Configuration structure
Posted by Niclas Hedhman <ni...@hedhman.org>.
Ok... I'll strip it out.
On Mon, Jun 5, 2017 at 3:41 PM, Paul Merlin <pa...@apache.org> wrote:
> Le 2017-06-04 16:09, Niclas Hedhman a écrit :
>
>> By that rationale, we should remove properties/xml/json/yaml files as
>> well,
>> right?
>>
>
> :)
>
> Yeah, maybe, maybe not.
>
> What I dislike in the added logic is that it exposes the application to
> the environment through a naming convention that does not involve structure
> nor service identity. Properties/xml/json/yaml at least need to be named
> according to the configured service identity. And I see them as almost
> immutable (classpath assembled at build time). For editable configuration
> defaults I rely on ad-hoc application configuration files applied at
> assembly time.
>
>
> I better see a SPI for configuration defaults. Extensions for this SPI
> could then use the environment, system properties, or anything else really.
> They could apply any named convention, or just pick some things from the
> environment.
>
> I then picture an implementation backed by e.g. Typesafe Config for nice
> configuration files.
> It supports using environment variables and system properties, but
> explicitly.
> One could then have a single `application.conf` file, here's a quick
> example:
>
> some_service_identity:
> aStringProperty = with some text
> anEnvironmentVariable = ${MY_ENV_VAR}
>
> another_service:
> somethingElse = 42
>
> Setting the `some_service_identity.aStringProperty` system property then
> override its value.
> See https://github.com/typesafehub/config
>
> Other implementations backed by etcd, zookeeper, consul etc.. could be
> provided for advanced deployments.
>
>
> All in all I'd be in favor of not adding any magic to 3.0 and think about
> a more holistic solution for 3.1.
>
> WDYT?
>
> Cheers
>
> /Paul
>
>
>
>
> On Sun, Jun 4, 2017 at 8:37 PM, Paul Merlin <pa...@apache.org> wrote:
>>
>> Le 2017-06-04 13:23, Niclas Hedhman a écrit :
>>>
>>> initialValueProvider should be marked "incomplete" or something.
>>>>
>>>>
>>> I'd rather simply remove it from the api for now.
>>> I'll do that if no one yell :)
>>>
>>>
>>> For the rest, not so sure I like it. I think I would prefer something
>>> more
>>>
>>>> composite-like, but not sure yet.
>>>>
>>>>
>>> One thing that is already possible is to declare defaults at assembly,
>>> getting system properties/env at that time. That's how I do for my apps
>>> and
>>> I like the explicitness.
>>>
>>> module.forMixins( MyConfig.class )
>>> .declareDefaults()
>>> .myProperty().set( System.getenv( "SOME_THING_WHATEVER_THE_NAME" )
>>> );
>>>
>>> Just to say, I'm not sure I like the extra magic.
>>> I'd favor it to be opt-in. But I'd be happy with a way to opt-out too.
>>>
>>
>
--
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java
Re: Configuration structure
Posted by Paul Merlin <pa...@apache.org>.
Le 2017-06-04 16:09, Niclas Hedhman a écrit :
> By that rationale, we should remove properties/xml/json/yaml files as
> well,
> right?
:)
Yeah, maybe, maybe not.
What I dislike in the added logic is that it exposes the application to
the environment through a naming convention that does not involve
structure nor service identity. Properties/xml/json/yaml at least need
to be named according to the configured service identity. And I see them
as almost immutable (classpath assembled at build time). For editable
configuration defaults I rely on ad-hoc application configuration files
applied at assembly time.
I better see a SPI for configuration defaults. Extensions for this SPI
could then use the environment, system properties, or anything else
really. They could apply any named convention, or just pick some things
from the environment.
I then picture an implementation backed by e.g. Typesafe Config for nice
configuration files.
It supports using environment variables and system properties, but
explicitly.
One could then have a single `application.conf` file, here's a quick
example:
some_service_identity:
aStringProperty = with some text
anEnvironmentVariable = ${MY_ENV_VAR}
another_service:
somethingElse = 42
Setting the `some_service_identity.aStringProperty` system property then
override its value.
See https://github.com/typesafehub/config
Other implementations backed by etcd, zookeeper, consul etc.. could be
provided for advanced deployments.
All in all I'd be in favor of not adding any magic to 3.0 and think
about a more holistic solution for 3.1.
WDYT?
Cheers
/Paul
> On Sun, Jun 4, 2017 at 8:37 PM, Paul Merlin <pa...@apache.org>
> wrote:
>
>> Le 2017-06-04 13:23, Niclas Hedhman a écrit :
>>
>>> initialValueProvider should be marked "incomplete" or something.
>>>
>>
>> I'd rather simply remove it from the api for now.
>> I'll do that if no one yell :)
>>
>>
>> For the rest, not so sure I like it. I think I would prefer something
>> more
>>> composite-like, but not sure yet.
>>>
>>
>> One thing that is already possible is to declare defaults at assembly,
>> getting system properties/env at that time. That's how I do for my
>> apps and
>> I like the explicitness.
>>
>> module.forMixins( MyConfig.class )
>> .declareDefaults()
>> .myProperty().set( System.getenv( "SOME_THING_WHATEVER_THE_NAME"
>> ) );
>>
>> Just to say, I'm not sure I like the extra magic.
>> I'd favor it to be opt-in. But I'd be happy with a way to opt-out too.
Re: Configuration structure
Posted by Niclas Hedhman <ni...@hedhman.org>.
By that rationale, we should remove properties/xml/json/yaml files as well,
right?
On Sun, Jun 4, 2017 at 8:37 PM, Paul Merlin <pa...@apache.org> wrote:
> Le 2017-06-04 13:23, Niclas Hedhman a écrit :
>
>> initialValueProvider should be marked "incomplete" or something.
>>
>
> I'd rather simply remove it from the api for now.
> I'll do that if no one yell :)
>
>
> For the rest, not so sure I like it. I think I would prefer something more
>> composite-like, but not sure yet.
>>
>
> One thing that is already possible is to declare defaults at assembly,
> getting system properties/env at that time. That's how I do for my apps and
> I like the explicitness.
>
> module.forMixins( MyConfig.class )
> .declareDefaults()
> .myProperty().set( System.getenv( "SOME_THING_WHATEVER_THE_NAME" ) );
>
> Just to say, I'm not sure I like the extra magic.
> I'd favor it to be opt-in. But I'd be happy with a way to opt-out too.
>
>
--
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java
Re: Configuration structure
Posted by Paul Merlin <pa...@apache.org>.
Le 2017-06-04 13:23, Niclas Hedhman a écrit :
> initialValueProvider should be marked "incomplete" or something.
I'd rather simply remove it from the api for now.
I'll do that if no one yell :)
> For the rest, not so sure I like it. I think I would prefer something
> more
> composite-like, but not sure yet.
One thing that is already possible is to declare defaults at assembly,
getting system properties/env at that time. That's how I do for my apps
and I like the explicitness.
module.forMixins( MyConfig.class )
.declareDefaults()
.myProperty().set( System.getenv( "SOME_THING_WHATEVER_THE_NAME" )
);
Just to say, I'm not sure I like the extra magic.
I'd favor it to be opt-in. But I'd be happy with a way to opt-out too.
Re: Configuration structure
Posted by Niclas Hedhman <ni...@hedhman.org>.
initialValueProvider should be marked "incomplete" or something.
For the rest, not so sure I like it. I think I would prefer something more
composite-like, but not sure yet.
On Sun, Jun 4, 2017 at 7:01 PM, Paul Merlin <pa...@apache.org> wrote:
> Maybe a silly idea, don't know, totally different, but ...
>
>
> interface MyConfig {
>
> @UseDefaults( value = "user.home", source =
> DefaultsSource.SYSTEM_PROPERTIES )
> Property<String> homeDirectoryPath();
>
> @UseDefaults( value = "SOME_PORT", source = DefaultsSource.ENVIRONMENT )
> Property<Integer> thePortWeWantToUse();
>
> Property<List<String>> somethingElse();
>
> }
>
> Too much annotations, erk ...
>
>
>
> Looking at it this way the goal could be seen as providing default values
> per property.
>
> We already have the assembly time initial value provider that can do just
> that, or could be enhanced to (couldn't find tests for it ...). We could
> then provide some helpers for system properties / environment variables
> mapping.
>
> BTW, shouldn't InitialValueProvider be named DefaultValueProvider instead,
> for consistency?
>
>
--
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java
Re: Configuration structure
Posted by Paul Merlin <pa...@apache.org>.
Maybe a silly idea, don't know, totally different, but ...
interface MyConfig {
@UseDefaults( value = "user.home", source =
DefaultsSource.SYSTEM_PROPERTIES )
Property<String> homeDirectoryPath();
@UseDefaults( value = "SOME_PORT", source = DefaultsSource.ENVIRONMENT
)
Property<Integer> thePortWeWantToUse();
Property<List<String>> somethingElse();
}
Too much annotations, erk ...
Looking at it this way the goal could be seen as providing default
values per property.
We already have the assembly time initial value provider that can do
just that, or could be enhanced to (couldn't find tests for it ...). We
could then provide some helpers for system properties / environment
variables mapping.
BTW, shouldn't InitialValueProvider be named DefaultValueProvider
instead, for consistency?
Re: Configuration structure
Posted by Paul Merlin <pa...@apache.org>.
Le 2017-06-04 12:21, Niclas Hedhman a écrit :
> Well, I implemented the simple/hacky way now.
>
> public interface MyConfig
> {
> Property<String> osName();
>
> Property<String> home();
>
> Property<String> path();
> }
>
>
> will now read "os.name" from System Properties and $HOME and
> $PATH/%PATH%
> of Environment variables.
>
> Additionally, System properties takes precedence over environment
> variables, so env variables can be overridden more easily.
This is to get defaults if the configuration could not be found in the
corresponding ES, correct?
So, the values gathered from system properties and environment variables
during that "first run" will then be persisted into the corresponding
ES. If the ES is persistent, then the next run of the application will
not use system properties nor environment variables but the values
fetched from the configuration ES, that is values captured from "the
environment" during the first run.
Of course, if using a non-persistent ES for configuration entities (e.g.
MemoryES), then "the environment" will be used on each run.
I didn't think this through but it could get surprising.
As a side note, system properties and environment variable are mutable.
So this new mechanism is in fact taking a copy of their values at some
point in time, application startup, or later service activation.
Re: Configuration structure
Posted by Niclas Hedhman <ni...@hedhman.org>.
Well, I implemented the simple/hacky way now.
public interface MyConfig
{
Property<String> osName();
Property<String> home();
Property<String> path();
}
will now read "os.name" from System Properties and $HOME and $PATH/%PATH%
of Environment variables.
Additionally, System properties takes precedence over environment
variables, so env variables can be overridden more easily.
Just pushed to 'develop' together with both yeoman stuff and more docs.
On Thu, Jun 1, 2017 at 11:02 AM, Niclas Hedhman <ni...@hedhman.org> wrote:
> By allowing the fallback to system properties and/or environment variables
> for configuration, it would be a lot easier to provide configuration from
> commandline. But the devil is in the details.
>
> 1. Should it be fallback for a single unset property?
> 2. Should it be only for complete missing initial properties file?
> 3. Should it be only for first initialization, or should it use it even
> if an Entity is present?
> 4. Should there be some kind of "name space" in the System properties,
> or should the "name" be enough?
>
>
> I think it would be pretty cool if one could simply do;
>
> public interface MyConfiguration
> {
> Property<String> osName();
> Property<String> userDir();
> }
>
> and get those directly populated from System properties.
>
> Hmmm... Could that be done via Lifecycle?
>
> No, because the validation check kicks in before the Lifecycle methods are
> called.
>
> Maybe, just maybe, all this special handling in Configuration should
> actual be pushed to some type of Fragment that gets a go at it before the
> validation is done. Because if we could do that generically, and have a
> compatible implementation for current Configuration behavior, I think we
> are much much better off, and the semantics are effectively in the hands of
> users.
>
>
>
> I think we push it to 3.1 since there are too many questions to be hashed
> out and not enough time to try it out.
>
> On Tue, May 30, 2017 at 1:23 AM, Kent Sølvsten <ke...@gmail.com>
> wrote:
>
>> I think it is a very good idea.
>>
>> As long as we retain an opt-out possibility to avoid accessing the
>> system-properties (reasons to avoid them could either be "pureness"
>> reasons
>> or the presence of a securitymanager).
>>
>> On Mon, May 29, 2017 at 9:33 AM, Niclas Hedhman <ni...@hedhman.org>
>> wrote:
>>
>> > The least intrusive implementation of this is to take out the exception
>> > thrown on org/apache/polygene/api/composite/PropertyMapper.java:117.
>> Then
>> > add the backup for the two maps mentioned above, with some
>> > strategy/priority.
>> >
>> > Niclas
>> >
>> > On Mon, May 29, 2017 at 1:47 PM, Niclas Hedhman <ni...@hedhman.org>
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > at the moment, the content of the properties file drives what is
>> expected
>> > > in the ConfigurationComposite. If there are more properties than there
>> > are
>> > > declared and matching Proprety<> methods, then there is an Exception.
>> > >
>> > > This might have been rational back in the days when this was discussed
>> > > first time, but if we are heading towards supporting external and
>> perhaps
>> > > more exotic configuration "supply-chains", then I think it would be
>> more
>> > > logical that the ConfigurationComposite simply reads what it wants and
>> > > ignores everything else.
>> > >
>> > > AND then would could have this super cool addition that if no files
>> are
>> > > found that works, fall back to System.getProperties() and
>> > System.getenv() as
>> > > the final backups.
>> > >
>> > > WDYT?
>> > >
>> > > Cheers
>> > > --
>> > > Niclas Hedhman, Software Developer
>> > > http://polygene.apache.org - New Energy for Java
>> > >
>> >
>> >
>> >
>> > --
>> > Niclas Hedhman, Software Developer
>> > http://polygene.apache.org - New Energy for Java
>> >
>>
>
>
>
> --
> Niclas Hedhman, Software Developer
> http://polygene.apache.org - New Energy for Java
>
--
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java
Re: Configuration structure
Posted by Niclas Hedhman <ni...@hedhman.org>.
By allowing the fallback to system properties and/or environment variables
for configuration, it would be a lot easier to provide configuration from
commandline. But the devil is in the details.
1. Should it be fallback for a single unset property?
2. Should it be only for complete missing initial properties file?
3. Should it be only for first initialization, or should it use it even
if an Entity is present?
4. Should there be some kind of "name space" in the System properties,
or should the "name" be enough?
I think it would be pretty cool if one could simply do;
public interface MyConfiguration
{
Property<String> osName();
Property<String> userDir();
}
and get those directly populated from System properties.
Hmmm... Could that be done via Lifecycle?
No, because the validation check kicks in before the Lifecycle methods are
called.
Maybe, just maybe, all this special handling in Configuration should actual
be pushed to some type of Fragment that gets a go at it before the
validation is done. Because if we could do that generically, and have a
compatible implementation for current Configuration behavior, I think we
are much much better off, and the semantics are effectively in the hands of
users.
I think we push it to 3.1 since there are too many questions to be hashed
out and not enough time to try it out.
On Tue, May 30, 2017 at 1:23 AM, Kent Sølvsten <ke...@gmail.com>
wrote:
> I think it is a very good idea.
>
> As long as we retain an opt-out possibility to avoid accessing the
> system-properties (reasons to avoid them could either be "pureness" reasons
> or the presence of a securitymanager).
>
> On Mon, May 29, 2017 at 9:33 AM, Niclas Hedhman <ni...@hedhman.org>
> wrote:
>
> > The least intrusive implementation of this is to take out the exception
> > thrown on org/apache/polygene/api/composite/PropertyMapper.java:117.
> Then
> > add the backup for the two maps mentioned above, with some
> > strategy/priority.
> >
> > Niclas
> >
> > On Mon, May 29, 2017 at 1:47 PM, Niclas Hedhman <ni...@hedhman.org>
> > wrote:
> >
> > > Hi,
> > >
> > > at the moment, the content of the properties file drives what is
> expected
> > > in the ConfigurationComposite. If there are more properties than there
> > are
> > > declared and matching Proprety<> methods, then there is an Exception.
> > >
> > > This might have been rational back in the days when this was discussed
> > > first time, but if we are heading towards supporting external and
> perhaps
> > > more exotic configuration "supply-chains", then I think it would be
> more
> > > logical that the ConfigurationComposite simply reads what it wants and
> > > ignores everything else.
> > >
> > > AND then would could have this super cool addition that if no files are
> > > found that works, fall back to System.getProperties() and
> > System.getenv() as
> > > the final backups.
> > >
> > > WDYT?
> > >
> > > Cheers
> > > --
> > > Niclas Hedhman, Software Developer
> > > http://polygene.apache.org - New Energy for Java
> > >
> >
> >
> >
> > --
> > Niclas Hedhman, Software Developer
> > http://polygene.apache.org - New Energy for Java
> >
>
--
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java
Re: Configuration structure
Posted by Kent Sølvsten <ke...@gmail.com>.
I think it is a very good idea.
As long as we retain an opt-out possibility to avoid accessing the
system-properties (reasons to avoid them could either be "pureness" reasons
or the presence of a securitymanager).
On Mon, May 29, 2017 at 9:33 AM, Niclas Hedhman <ni...@hedhman.org> wrote:
> The least intrusive implementation of this is to take out the exception
> thrown on org/apache/polygene/api/composite/PropertyMapper.java:117. Then
> add the backup for the two maps mentioned above, with some
> strategy/priority.
>
> Niclas
>
> On Mon, May 29, 2017 at 1:47 PM, Niclas Hedhman <ni...@hedhman.org>
> wrote:
>
> > Hi,
> >
> > at the moment, the content of the properties file drives what is expected
> > in the ConfigurationComposite. If there are more properties than there
> are
> > declared and matching Proprety<> methods, then there is an Exception.
> >
> > This might have been rational back in the days when this was discussed
> > first time, but if we are heading towards supporting external and perhaps
> > more exotic configuration "supply-chains", then I think it would be more
> > logical that the ConfigurationComposite simply reads what it wants and
> > ignores everything else.
> >
> > AND then would could have this super cool addition that if no files are
> > found that works, fall back to System.getProperties() and
> System.getenv() as
> > the final backups.
> >
> > WDYT?
> >
> > Cheers
> > --
> > Niclas Hedhman, Software Developer
> > http://polygene.apache.org - New Energy for Java
> >
>
>
>
> --
> Niclas Hedhman, Software Developer
> http://polygene.apache.org - New Energy for Java
>
Re: Configuration structure
Posted by Niclas Hedhman <ni...@hedhman.org>.
The least intrusive implementation of this is to take out the exception
thrown on org/apache/polygene/api/composite/PropertyMapper.java:117. Then
add the backup for the two maps mentioned above, with some
strategy/priority.
Niclas
On Mon, May 29, 2017 at 1:47 PM, Niclas Hedhman <ni...@hedhman.org> wrote:
> Hi,
>
> at the moment, the content of the properties file drives what is expected
> in the ConfigurationComposite. If there are more properties than there are
> declared and matching Proprety<> methods, then there is an Exception.
>
> This might have been rational back in the days when this was discussed
> first time, but if we are heading towards supporting external and perhaps
> more exotic configuration "supply-chains", then I think it would be more
> logical that the ConfigurationComposite simply reads what it wants and
> ignores everything else.
>
> AND then would could have this super cool addition that if no files are
> found that works, fall back to System.getProperties() and System.getenv() as
> the final backups.
>
> WDYT?
>
> Cheers
> --
> Niclas Hedhman, Software Developer
> http://polygene.apache.org - New Energy for Java
>
--
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java