You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sling.apache.org by Andreas Schaefer <sc...@me.com.INVALID> on 2018/07/06 20:59:55 UTC
/etc/map with Placeholders for farms / dev stack
Hi
I create a ticket asking for the addition of placeholder support in /etc/map:
https://issues.apache.org/jira/browse/SLING-7768 <https://issues.apache.org/jira/browse/SLING-7768>
The OSGi Web Plugin for the Resource Resolver actually shows the resolved values making it easy to understand and check the /etc/map entries.
For the Placeholder provider I also think it would be great if the placeholder values could be provided from a URL (file or web page) and also as CLI arguments:
java -jar sling*.jar -Dphv.fq.host.name=my-host.com <http://my-host.com/>
This would be helpful if a user messed up the /etc/map and cannot get back into Sling.
Let me know what you think.
Cheers - Andreas Schaefer
RE: Configuration property sources with Sling
Posted by Stefan Seifert <ss...@pro-vision.de>.
>Now Jörg suggested the context-aware configuration override mechanism
>[2]
>which makes sense except that all relevant configuration parts
>like distribution/replication agents, externaliser and other OSGi
>configurations (e.g. for jdbc db configs or search engine end points)
>are not configured via caconfig today.
and caconfig was never intended as a replacement for system configuration, but only as an addition for context-aware configuration which cannot be managed elegantly as OSGi configurations. most sensitive configurations like jdbc configs etc. are system configurations and in most cases not context-aware.
stefan
Re: Configuration property sources with Sling
Posted by Georg Henzler <sl...@ghenzler.de>.
Hi Robert,
>> * An OSGi configuration
>> * A local properties file
>> * OS Env variables
>> * Java System Properties
>
> Bringing this out a new thread as I think it will slip by otherwise.
>
> IIUC, you suggest that we extend the Sling configuration mechanism by
> making other sources first-class besides OSGi. What are the use cases
> for that? Currently we delegate almost everything to OSGi component
> configurations, with some settings falling back or reading from
> framework properties ( which can also fall back to system properties ).
The use cases are described in [1] (first part). At the moment there are
two common config sources for Sling: OSGi configs and content resources
(/etc/maps, rewriter, context-aware configs, etc.). My suggestion is
exactly to not introduce more sources, but rather have for very few
uses cases given in [1] (Hosts/ports, admin passwords) the ability to
"pre-process" the configs before they become visible+effective to Sling.
Only for that pre-processing I would love to see the four options from
above to start with (best case with an SPI for extensibility, and best
case without changing all code that should be able to consume it).
Now Jörg suggested the context-aware configuration override mechanism
[2]
which makes sense except that all relevant configuration parts
like distribution/replication agents, externaliser and other OSGi
configurations (e.g. for jdbc db configs or search engine end points)
are not configured via caconfig today.
-Georg
[1]
http://jackrabbit.510166.n4.nabble.com/Environment-specific-non-role-based-configurations-td4669795.html
[2]
http://sling.apache.org/documentation/bundles/context-aware-configuration/context-aware-configuration-override.html
Configuration property sources with Sling (was: /etc/map with
Placeholders for farms / dev stack)
Posted by Robert Munteanu <ro...@apache.org>.
Hi,
On Tue, 2018-07-31 at 01:29 +0200, Georg Henzler wrote:
> A property file should still be possible... so if we do this
> in Sling, I think the following is needed:
>
> Property Sources to be supported at minimum:
>
> * An OSGi configuration
> * A local properties file
> * OS Env variables
> * Java System Properties
Bringing this out a new thread as I think it will slip by otherwise.
IIUC, you suggest that we extend the Sling configuration mechanism by
making other sources first-class besides OSGi. What are the use cases
for that? Currently we delegate almost everything to OSGi component
configurations, with some settings falling back or reading from
framework properties ( which can also fall back to system properties ).
Thanks,
Robert
Re: /etc/map with Placeholders for farms / dev stack
Posted by Andreas Schaefer <sc...@me.com.INVALID>.
Hi George
> On Jul 30, 2018, at 4:29 PM, Georg Henzler <sl...@ghenzler.de> wrote:
>
> Hi Andreas,
>
>> Install Hooks have their own issues like that they do not work for
>> replication
>
> Install Hooks work flawlessly with replication since around two years
> (AEM 6.3)
Cool. I probably ran into that a little bit before that.
>
>>> ... allowing external URLs is dangerous...
>> I do agree with that and I will remove that from my POC.
>
> A property file should still be possible... so if we do this
> in Sling, I think the following is needed:
>
> Property Sources to be supported at minimum:
>
> * An OSGi configuration
> * A local properties file
> * OS Env variables
> * Java System Properties
All but the local properties file I wanted to support. I cannot tell about the security implication of allowing a local properties file to avoid hackers to inject code.
>
> There should be an Felix console plugin to show what is active
> from what source and where the value is used. It should live in
> a bundle named like e.g. org.apache.sling.systemenv and provide
> a simple interface to perform string interpolation on any
> string. This service can then be used by
>
> * Resource resolver
> * Sling distribution
> * Context-Aware Configuration
The StringInterpolationProvider should not reside inside the Resource Resolver but in its own or a commons package. For my POC I just kept it there for convenience.
>
> a product like AEM can also use this interface in
>
> * AEM replication config
> * externaliser config
>
> To have this mechanism properly rolled out it will take some
> time (until all listed modules properly use the to-defined
> interface)
>
> IMHO it is not an option to do it locally in resourceresolver
> because
I started with the etc-mapping as this was the initial use case to move the ball forward.
>
> a) it would result in quite a bit of duplicated code across the
> modules listed above (once we actually start implement it)
> b) if additional sources need to be added in the future (think
> ZooKeeper as one interesting option to receive those env-specific
> values), all consumers would have to be updated (unlikely to
> happen, more likely is to have inconsistent behaviour over time)
The String Interpolation should be not be in the Resource Resolver and hence can be easily used from other modules. Creating a separate module / git repo for just one class this seems to be a little bit of an overkill. Is there any place where we can place that?
>
> Since the described above is quite a bit of effort, I opt for
> pushing a lower-level approach forward (something like [1] or
> maybe even something on oak level). Then there is only one
> implementation needed and it's available immediately, everywhere.
The ticket below is closed with ‘won’t fix’. So I am not sure if Jackrabbit wants to do that.
>
> -Georg
>
> [1] https://issues.apache.org/jira/browse/JCRVLT-254
>
>
>
Re: /etc/map with Placeholders for farms / dev stack
Posted by Georg Henzler <sl...@ghenzler.de>.
Hi Ruben,
> ... may not be the best solution as it would imply a package install
> every time a change to the configs are necessary
Updating upon changes is an whole own challenge for your approach -
if resource change listeners only react on changes of placeholder
strings (e.g. 'mystr-${myVar}') because that's the only thing saved
in the JCR and replacement is done across various Sling modules,
there needs to be a new mechanism to listen to changes to property
changes (for the case the value of ${myVar} changes in the previous
example).
> Not sure if maybe this would need a bigger spec to continue the work
> and alignment on what everybody would like to see
I think this would be good. Maybe best to collect goals and non-goals
on a wiki page at [1]
-Georg
[1] https://cwiki.apache.org/confluence/display/SLING
Re: /etc/map with Placeholders for farms / dev stack
Posted by Ruben Reusser <rr...@headwire.com>.
Georg,
I think this is a good discussion - looks like the sling team has to
make some decisions here and I am glad the etc/map work Andreas did is
at least sparking a bigger discussion.
From my point of view using packages and install hooks may work but may
not be the best solution as it would imply a package install every time
a change to the configs are necessary.
Not sure if maybe this would need a bigger spec to continue the work and
alignment on what everybody would like to see
Ruben
On 7/30/2018 4:29 PM, Georg Henzler wrote:
> Hi Andreas,
>
>> Install Hooks have their own issues like that they do not work for
>> replication
>
> Install Hooks work flawlessly with replication since around two years
> (AEM 6.3)
>
>> and they requires to changes whatever the customer has /
>> provides in order to add the place holders.
>
> not sure what you mean with this?
>
>>> ... allowing external URLs is dangerous...
>> I do agree with that and I will remove that from my POC.
>
> A property file should still be possible... so if we do this
> in Sling, I think the following is needed:
>
> Property Sources to be supported at minimum:
>
> * An OSGi configuration
> * A local properties file
> * OS Env variables
> * Java System Properties
>
> There should be an Felix console plugin to show what is active
> from what source and where the value is used. It should live in
> a bundle named like e.g. org.apache.sling.systemenv and provide
> a simple interface to perform string interpolation on any
> string. This service can then be used by
>
> * Resource resolver
> * Sling distribution
> * Context-Aware Configuration
>
> a product like AEM can also use this interface in
>
> * AEM replication config
> * externaliser config
>
> To have this mechanism properly rolled out it will take some
> time (until all listed modules properly use the to-defined
> interface)
>
> IMHO it is not an option to do it locally in resourceresolver
> because
>
> a) it would result in quite a bit of duplicated code across the
> modules listed above (once we actually start implement it)
> b) if additional sources need to be added in the future (think
> ZooKeeper as one interesting option to receive those env-specific
> values), all consumers would have to be updated (unlikely to
> happen, more likely is to have inconsistent behaviour over time)
>
> Since the described above is quite a bit of effort, I opt for
> pushing a lower-level approach forward (something like [1] or
> maybe even something on oak level). Then there is only one
> implementation needed and it's available immediately, everywhere.
>
> -Georg
>
> [1] https://issues.apache.org/jira/browse/JCRVLT-254
>
>
>
Re: /etc/map with Placeholders for farms / dev stack
Posted by Georg Henzler <sl...@ghenzler.de>.
Hi Andreas,
> Install Hooks have their own issues like that they do not work for
> replication
Install Hooks work flawlessly with replication since around two years
(AEM 6.3)
> and they requires to changes whatever the customer has /
> provides in order to add the place holders.
not sure what you mean with this?
>> ... allowing external URLs is dangerous...
> I do agree with that and I will remove that from my POC.
A property file should still be possible... so if we do this
in Sling, I think the following is needed:
Property Sources to be supported at minimum:
* An OSGi configuration
* A local properties file
* OS Env variables
* Java System Properties
There should be an Felix console plugin to show what is active
from what source and where the value is used. It should live in
a bundle named like e.g. org.apache.sling.systemenv and provide
a simple interface to perform string interpolation on any
string. This service can then be used by
* Resource resolver
* Sling distribution
* Context-Aware Configuration
a product like AEM can also use this interface in
* AEM replication config
* externaliser config
To have this mechanism properly rolled out it will take some
time (until all listed modules properly use the to-defined
interface)
IMHO it is not an option to do it locally in resourceresolver
because
a) it would result in quite a bit of duplicated code across the
modules listed above (once we actually start implement it)
b) if additional sources need to be added in the future (think
ZooKeeper as one interesting option to receive those env-specific
values), all consumers would have to be updated (unlikely to
happen, more likely is to have inconsistent behaviour over time)
Since the described above is quite a bit of effort, I opt for
pushing a lower-level approach forward (something like [1] or
maybe even something on oak level). Then there is only one
implementation needed and it's available immediately, everywhere.
-Georg
[1] https://issues.apache.org/jira/browse/JCRVLT-254
Re: /etc/map with Placeholders for farms / dev stack
Posted by Andreas Schaefer <sc...@me.com.INVALID>.
Hi George
> On Jul 26, 2018, at 2:21 PM, Georg Henzler <sl...@ghenzler.de> wrote:
>
> Hi Andreas,
>
> I'm not sure if this is going in the right direction - this is currently
> implemented at at a local module level (sling-org-apache-sling-resourceresolver)
> when it is really something that the platform should support (see below).
Ruben and I were discussing to add placeholders also to OSGi Configurations to be able to change host names for both the etc-mapping but also for Distribution / Replication.
That would requires to add the string interpolation to Felix in order to make it work as just trying to handle the OSG configuration from JCR leads to many issues.
But for now we would love to see this in Sling as soon as possible as testing of bigger companies is difficult and time consuming.
If we have a solution in Jackrabbit I am more than happy to remove my code from the resource resolver.
>
> Also note that your example/reasoning in the description of SLING-7768
> only lists DEV/QA/STAGING/PROD - if you are limited to those environments you
> can easily do with runmodes. Placeholder replacement becomes interesting when
> you think about an "elastic number of environments" that are not known at
> development time (think about spinning up "STAGING-7" for that special performance
> test that needs to run in isolation)
This was the basic test case but our case this involves to have a different host name because of a temporary migration server.
It would also work for URL paths as well.
>
>> ... Ruben and I think it could be helpful to make the Placeholder Provider
>> more generic to that it could be used in other services like the
>> distribution/replication, externalizers etc to have an easier way to
>> manage server configuration in a central place.
>
> I agree we should have the capability in the platform, but IMHO it should not
> be in sling-org-apache-sling-resourceresolver. There was a discussion in
> jackrabbit-dev:
>
> http://jackrabbit.510166.n4.nabble.com/Environment-specific-non-role-based-configurations-td4669795.html
> and https://issues.apache.org/jira/browse/JCRVLT-254
> for the initial issue.
>
> Btw.: you can achieve your requirement already today (down to AEM 6.0) with
> https://github.com/Netcentric/system-env-install-hook
> (but I'd rather see this as first-class citizen as part of the platform
> and not as a third-party module)
Install Hooks have their own issues like that they do not work for replication and they requires to changes whatever the customer has / provides in order to add the place holders.
>
>>> * No need to support anything else like files and system properties to
>>> provide the placeholders. Makes error handling much more complicated (what
>>> do you do if the http download of that placeholder file fails?), and I
>>> would rather leave it to the ops team to implement that process and handle
>>> any errors.
>> I see the issue with the external files / web resources. The system
>> properties are there to make sure an admin can override any settings
>> if things do not work out.
>
> I agree here with Jörg that allowing external URLs is dangerous here (and
> it could even be seen critical from a security perspective). Also, the config allows
> to give both key value pairs directly and a URL to retrieve them from - I suppose
> these two property sets are merged?
I do agree with that and I will remove that from my POC.
>
> But to sum up I think the best would be to have a mechanism for placeholder replacement
> on a lower level...
>
> -Georg
Re: /etc/map with Placeholders for farms / dev stack
Posted by Georg Henzler <sl...@ghenzler.de>.
Hi Andreas,
I'm not sure if this is going in the right direction - this is currently
implemented at at a local module level
(sling-org-apache-sling-resourceresolver)
when it is really something that the platform should support (see
below).
Also note that your example/reasoning in the description of SLING-7768
only lists DEV/QA/STAGING/PROD - if you are limited to those
environments you
can easily do with runmodes. Placeholder replacement becomes interesting
when
you think about an "elastic number of environments" that are not known
at
development time (think about spinning up "STAGING-7" for that special
performance
test that needs to run in isolation)
> ... Ruben and I think it could be helpful to make the Placeholder
> Provider
> more generic to that it could be used in other services like the
> distribution/replication, externalizers etc to have an easier way to
> manage server configuration in a central place.
I agree we should have the capability in the platform, but IMHO it
should not
be in sling-org-apache-sling-resourceresolver. There was a discussion in
jackrabbit-dev:
http://jackrabbit.510166.n4.nabble.com/Environment-specific-non-role-based-configurations-td4669795.html
and https://issues.apache.org/jira/browse/JCRVLT-254
for the initial issue.
Btw.: you can achieve your requirement already today (down to AEM 6.0)
with
https://github.com/Netcentric/system-env-install-hook
(but I'd rather see this as first-class citizen as part of the platform
and not as a third-party module)
>> * No need to support anything else like files and system properties to
>> provide the placeholders. Makes error handling much more complicated
>> (what
>> do you do if the http download of that placeholder file fails?), and I
>> would rather leave it to the ops team to implement that process and
>> handle
>> any errors.
>
> I see the issue with the external files / web resources. The system
> properties are there to make sure an admin can override any settings
> if things do not work out.
I agree here with Jörg that allowing external URLs is dangerous here
(and
it could even be seen critical from a security perspective). Also, the
config allows
to give both key value pairs directly and a URL to retrieve them from -
I suppose
these two property sets are merged?
But to sum up I think the best would be to have a mechanism for
placeholder replacement
on a lower level...
-Georg
Re: /etc/map with Placeholders for farms / dev stack
Posted by Carsten Ziegeler <cz...@apache.org>.
Hi,
I only had a brief look at the current code, and currently it is
exporting the package org.apache.sling.resourceresolver.impl.mapping
which looks wrong to me. If we need new API this should go into the API
bundle. But I would prefer if we can get away without a new interface
Carsten
Andreas Schaefer wrote
> The placeholder was renamed to sling interpolation, changed the delimiter to ${} and added a jUnit test suite where the StringInterpolationProvider replacement is tested through Map Entries.
>
> I am wondering if Sling has some Integration Tests (IT). I could not find any further tests regarding /etc/map.
>
> Cheers - Andreas Schaefer
>
>> On Jul 12, 2018, at 1:15 AM, Bertrand Delacretaz <bd...@apache.org> wrote:
>>
>> Hi,
>>
>> On Mon, Jul 9, 2018 at 4:18 PM Andreas Schaefer
>> <sc...@me.com.invalid> wrote:
>>> ...Ruben and I think it could be helpful to make the Placeholder Provider more generic...
>>
>> I agree with that and I think "string interpolation" is a better name
>> than "placeholder" which is quite generic - as per
>> https://en.wikipedia.org/wiki/String_interpolation
>>
>> -Bertrand
>
--
Carsten Ziegeler
Adobe Research Switzerland
cziegeler@apache.org
Re: /etc/map with Placeholders for farms / dev stack
Posted by Bertrand Delacretaz <bd...@apache.org>.
Hi,
On Thu, Jul 26, 2018 at 12:51 AM Andreas Schaefer
<sc...@me.com.invalid> wrote:
> ...I am wondering if Sling has some Integration Tests (IT). I could not find any further tests regarding /etc/map. ..
Our "global" integration tests are at
https://github.com/apache/sling-org-apache-sling-launchpad-integration-tests
and https://github.com/apache/sling-org-apache-sling-launchpad-test-services
but over time we've been trying to bring those closer to the code that
they test, so many modules have ITs directly in their codebase.
I suspect we might be lacking such tests for /etc/map, there's very
few test files there which have "map" in their name, and no files
under src which contain the string "etc/map".
-Bertrand
Re: /etc/map with Placeholders for farms / dev stack
Posted by Andreas Schaefer <sc...@me.com.INVALID>.
The placeholder was renamed to sling interpolation, changed the delimiter to ${} and added a jUnit test suite where the StringInterpolationProvider replacement is tested through Map Entries.
I am wondering if Sling has some Integration Tests (IT). I could not find any further tests regarding /etc/map.
Cheers - Andreas Schaefer
> On Jul 12, 2018, at 1:15 AM, Bertrand Delacretaz <bd...@apache.org> wrote:
>
> Hi,
>
> On Mon, Jul 9, 2018 at 4:18 PM Andreas Schaefer
> <sc...@me.com.invalid> wrote:
>> ...Ruben and I think it could be helpful to make the Placeholder Provider more generic...
>
> I agree with that and I think "string interpolation" is a better name
> than "placeholder" which is quite generic - as per
> https://en.wikipedia.org/wiki/String_interpolation
>
> -Bertrand
Re: /etc/map with Placeholders for farms / dev stack
Posted by Bertrand Delacretaz <bd...@apache.org>.
Hi,
On Mon, Jul 9, 2018 at 4:18 PM Andreas Schaefer
<sc...@me.com.invalid> wrote:
> ...Ruben and I think it could be helpful to make the Placeholder Provider more generic...
I agree with that and I think "string interpolation" is a better name
than "placeholder" which is quite generic - as per
https://en.wikipedia.org/wiki/String_interpolation
-Bertrand
Re: /etc/map with Placeholders for farms / dev stack
Posted by Andreas Schaefer <sc...@me.com.INVALID>.
> On Jul 7, 2018, at 4:26 AM, Jörg Hoh <jh...@googlemail.com.INVALID> wrote:
>
> Hi Andreas,
>
> 2018-07-06 20:59 GMT+00:00 Andreas Schaefer <sc...@me.com.invalid>:
>
>>
>>
>> https://issues.apache.org/jira/browse/SLING-7768 <
>> https://issues.apache.org/jira/browse/SLING-7768>
>>
>
>
> In this ticket I see that you allow not only the replacement of property
> values, but also of resource names. Is that required? IIRC you can also
> work with "sling:match" properties, so the name of the resource is ignored.
> You haven't posted any implementation yet, so I wonder if that limitation
> could reduce the amount of changes to actually implement it.
I just posted a branch to the org-apache-sling-resourceresolver module called feature/SLING-7768.
I think it would be best to support both ways (the node name as well as the sling:match) to make it easier on the user.
Ruben and I think it could be helpful to make the Placeholder Provider more generic to that it could be used in other services like the distribution/replication, externalizers etc to have an easier way to manage server configuration in a central place.
>
>
>>
>> The OSGi Web Plugin for the Resource Resolver actually shows the resolved
>> values making it easy to understand and check the /etc/map entries.
>>
>> For the Placeholder provider I also think it would be great if the
>> placeholder values could be provided from a URL (file or web page) and also
>> as CLI arguments:
>>
>> java -jar sling*.jar -Dphv.fq.host.name=my-host.com <http://my-host.com/>
>>
>> This would be helpful if a user messed up the /etc/map and cannot get back
>> into Sling.
>>
>
> Nice feature. My whishlist for it:
>
> * When I provide the placeholders via a file, this file should be monitored
> and reloaded as I change it. If I add more /etc/map entries during runtime
> and need to introduce new placeholders, I would like to update the
> placeholder file in parallel, and have the new entries effective without
> restarting.
> * No need to support anything else like files and system properties to
> provide the placeholders. Makes error handling much more complicated (what
> do you do if the http download of that placeholder file fails?), and I
> would rather leave it to the ops team to implement that process and handle
> any errors.
I see the issue with the external files / web resources. The system properties are there to make sure an admin can override any settings if things do not work out.
> * Good test coverage (the resource resolver grew into a complex beast over
> the years…)
Sure, will do.
> * I would prefer the "${placeholder}" format instead of "{{{placeholder}}}"
> format :-)
Do JCR node names support ${}? If so that can be easily changed.
>
>
> --
> Cheers,
> Jörg Hoh,
>
> http://cqdump.wordpress.com
> Twitter: @joerghoh
Re: /etc/map with Placeholders for farms / dev stack
Posted by Jörg Hoh <jh...@googlemail.com.INVALID>.
Hi Andreas,
2018-07-06 20:59 GMT+00:00 Andreas Schaefer <sc...@me.com.invalid>:
>
>
> https://issues.apache.org/jira/browse/SLING-7768 <
> https://issues.apache.org/jira/browse/SLING-7768>
>
In this ticket I see that you allow not only the replacement of property
values, but also of resource names. Is that required? IIRC you can also
work with "sling:match" properties, so the name of the resource is ignored.
You haven't posted any implementation yet, so I wonder if that limitation
could reduce the amount of changes to actually implement it.
>
> The OSGi Web Plugin for the Resource Resolver actually shows the resolved
> values making it easy to understand and check the /etc/map entries.
>
> For the Placeholder provider I also think it would be great if the
> placeholder values could be provided from a URL (file or web page) and also
> as CLI arguments:
>
> java -jar sling*.jar -Dphv.fq.host.name=my-host.com <http://my-host.com/>
>
> This would be helpful if a user messed up the /etc/map and cannot get back
> into Sling.
>
Nice feature. My whishlist for it:
* When I provide the placeholders via a file, this file should be monitored
and reloaded as I change it. If I add more /etc/map entries during runtime
and need to introduce new placeholders, I would like to update the
placeholder file in parallel, and have the new entries effective without
restarting.
* No need to support anything else like files and system properties to
provide the placeholders. Makes error handling much more complicated (what
do you do if the http download of that placeholder file fails?), and I
would rather leave it to the ops team to implement that process and handle
any errors.
* Good test coverage (the resource resolver grew into a complex beast over
the years...)
* I would prefer the "${placeholder}" format instead of "{{{placeholder}}}"
format :-)
--
Cheers,
Jörg Hoh,
http://cqdump.wordpress.com
Twitter: @joerghoh