You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sling.apache.org by Ian Boston <ie...@tfd.co.uk> on 2013/10/08 19:40:10 UTC
TopologyConnectorServlet
Hi,
The whitelist configuration in this servlet is causing some problems
where the contents of the whitelist is potentially large and changing,
since it requires constant re-configuration.
Would it be possible to have a API service that is consulted if
present to check if the request is allowed. For those that want to use
the service they would configure the whitelist to reject everything
while the service was not present so avoid startup issues.
eg
+ @Reference(cardinality=ReferenceCardinality.OPTIONAL_UNARY)
+ private WhiteListProvider whiteListProvider;
/** Checks if the provided request's remote server is whitelisted **/
private boolean isWhitelisted(final SlingHttpServletRequest request) {
+ if (whiteListProvider != null) {
+ whiteListProvider.isWhitelisted(request);
+ }
if (whitelist.contains(request.getRemoteAddr())) {
return true;
} else if (whitelist.contains(request.getRemoteHost())) {
return true;
}
logger.info("isWhitelisted: rejecting " + request.getRemoteAddr()
+ ", " + request.getRemoteHost());
return false;
}
and in the API, presumably discovery api.
public interface WhiteListProvider {
private boolean isWhitelisted(HttpServletRequest request);
}
Best Regards
Ian
Re: TopologyConnectorServlet
Posted by Ian Boston <ie...@tfd.co.uk>.
On 10 October 2013 05:40, Felix Meschberger <fm...@adobe.com> wrote:
> Hi
>
> Am 09.10.2013 um 09:05 schrieb Ian Boston:
>
>> Hi,
>>
>> On 9 October 2013 16:21, Felix Meschberger <fm...@adobe.com> wrote:
>>> Hi
>>>
>>> Am 08.10.2013 um 10:40 schrieb Ian Boston:
>>>
>>>> Hi,
>>>> The whitelist configuration in this servlet is causing some problems
>>>> where the contents of the whitelist is potentially large and changing,
>>>> since it requires constant re-configuration.
>>>
>>> What is the problem with that ?
>>
>> When the whitelist in the central discovery endpoint depends on the
>> topology itself, it causes production deployment problems, especially
>> when the configuration of the end point and the management of the
>> topology of the production deployment are managed by different roles
>> within the organisation. This is typical of many organisations.
>> Engineers don't manage operations and TechOps don't manage
>> engineering. Normally configuration would be managed by TechOps but
>
> Fully agreed ! Which is exactly the reason why conceptually we have to separate development from deployment and which is why configuration *is* in fact a deployment issue and not a development issue.
>
>> with Sling that doesn't appear to be the case.
>
> How come ? That is not the intent of how Sling should be used.
>
Many systems that are managed by TechOps are configured by files on
disk (eg Apache httpd.conf etc). As a result much of the tooling to
manage production environments works best managing those files. Things
that are not files on disk, are generally considered part of the
application, unless the application itself is a TechOps application
(eg Ldap).
Sling (and tbh many Java applications) puts its configuration into
database files or repositories, accessed only once the application is
running which means TechOps cant use standard tools (Puppet, Cactus
etc) to manage the application. TechOps have to use application level
tooling and build systems (mvn) to interact with the configuration or
take undocumented routes to configuring the application (eg
manipulating the private Felix configuration files).
Because OSGi makes it easy to configure components, applications based
on Sling have large volumes of configuration, much of which is not so
much about deployment, and more about configuring the application. In
real life, that tends to be poorly documented and the effort level
required for TechOps to write build, deploy and maintain those
configuration files is too high. Hence the developer teams provide
them as part of application startup for TechOps to deploy.
There is nothing wrong with this, provided the configuration does not
depend on the deployment topology. When it does, it falls between the
gaps.
There are 2 potential solutions:
A. Create something along the lines of Puppet or Cactus to bring
deployment level Sling configuration into line with those systems. (or
whatever the chosen deployment tooling is).
B. Ensure that configuration performed in Sling is not dependent on
the deployment architecture and topology so that 99% of the
configuration can be used unchanged when deploying 10 or 10000
instances.
In this case, B is a lot less effort than A. A is not impossible, but
it is not as simple as writing a puppet module with a few curl
operations, for a start, deployment should be achievable with the
application not running.
In short, *if* all configuration is a deployment issue, then it should
work with existing deployment tooling.
Best Regards
Ian
Re: TopologyConnectorServlet
Posted by Felix Meschberger <fm...@adobe.com>.
Hi
Am 09.10.2013 um 09:05 schrieb Ian Boston:
> Hi,
>
> On 9 October 2013 16:21, Felix Meschberger <fm...@adobe.com> wrote:
>> Hi
>>
>> Am 08.10.2013 um 10:40 schrieb Ian Boston:
>>
>>> Hi,
>>> The whitelist configuration in this servlet is causing some problems
>>> where the contents of the whitelist is potentially large and changing,
>>> since it requires constant re-configuration.
>>
>> What is the problem with that ?
>
> When the whitelist in the central discovery endpoint depends on the
> topology itself, it causes production deployment problems, especially
> when the configuration of the end point and the management of the
> topology of the production deployment are managed by different roles
> within the organisation. This is typical of many organisations.
> Engineers don't manage operations and TechOps don't manage
> engineering. Normally configuration would be managed by TechOps but
Fully agreed ! Which is exactly the reason why conceptually we have to separate development from deployment and which is why configuration *is* in fact a deployment issue and not a development issue.
> with Sling that doesn't appear to be the case.
How come ? That is not the intent of how Sling should be used.
> Consequently the 2 get
> out of sync, and if you are using Sling Discovery to transport
> metadata about your topology, lots of things start to break when nodes
> are not in the whitelist.
Sure, if developers want to be involved with deployment, things run havock.
>
> In our special case we have 1 central discovery end point, if we had
> n, the problem would multiply by n.
>
>>
>> And why would a service API help mitigate that issue ?
>
> It would give those that are able to see the deployment infrastructure
> that they are dealing with, the opportunity to address this issue.
>
>> Considering that service would have to be reconfigured, too, and your code sample belows uses a static reference ?
>
> AFAIK it uses an optional reference, satisfied by a implementation
> outside Sling. The snippet was written from memory directly into the
> email to communicate what I was trying to discuss. Its is not real
> code.
>
>>
>> Also, noted in another thread, this would require export of the API from the impl bundle, which I am not very happy with...
>>
>> Finally -- and this is purely subjective -- I am not very happy with the name WhiteListProvider and then WhiteListProvider.isWhiteListed... Also would there be other needs for hooking into this servlet ?
>>
>
> I have been working on an implementation most of the day that is
> nothing like what was first discussed, I will commit in a moment. Much
> better to share real code with unit test coverage.
Ok.
Regards
Felix
>
> Best Regards
> Ian
>
>
>> Regards
>> Felix
>>
>>
>>
>>>
>>> Would it be possible to have a API service that is consulted if
>>> present to check if the request is allowed. For those that want to use
>>> the service they would configure the whitelist to reject everything
>>> while the service was not present so avoid startup issues.
>>>
>>> eg
>>>
>>> + @Reference(cardinality=ReferenceCardinality.OPTIONAL_UNARY)
>>> + private WhiteListProvider whiteListProvider;
>>>
>>> /** Checks if the provided request's remote server is whitelisted **/
>>>
>>> private boolean isWhitelisted(final SlingHttpServletRequest request) {
>>> + if (whiteListProvider != null) {
>>> + whiteListProvider.isWhitelisted(request);
>>> + }
>>> if (whitelist.contains(request.getRemoteAddr())) {
>>>
>>> return true;
>>>
>>> } else if (whitelist.contains(request.getRemoteHost())) {
>>>
>>> return true;
>>>
>>> }
>>>
>>> logger.info("isWhitelisted: rejecting " + request.getRemoteAddr()
>>>
>>> + ", " + request.getRemoteHost());
>>>
>>> return false;
>>>
>>> }
>>>
>>>
>>> and in the API, presumably discovery api.
>>>
>>> public interface WhiteListProvider {
>>>
>>> private boolean isWhitelisted(HttpServletRequest request);
>>>
>>> }
>>>
>>>
>>> Best Regards
>>> Ian
>>
Re: TopologyConnectorServlet
Posted by Ian Boston <ie...@tfd.co.uk>.
Hi,
On 9 October 2013 16:21, Felix Meschberger <fm...@adobe.com> wrote:
> Hi
>
> Am 08.10.2013 um 10:40 schrieb Ian Boston:
>
>> Hi,
>> The whitelist configuration in this servlet is causing some problems
>> where the contents of the whitelist is potentially large and changing,
>> since it requires constant re-configuration.
>
> What is the problem with that ?
When the whitelist in the central discovery endpoint depends on the
topology itself, it causes production deployment problems, especially
when the configuration of the end point and the management of the
topology of the production deployment are managed by different roles
within the organisation. This is typical of many organisations.
Engineers don't manage operations and TechOps don't manage
engineering. Normally configuration would be managed by TechOps but
with Sling that doesn't appear to be the case. Consequently the 2 get
out of sync, and if you are using Sling Discovery to transport
metadata about your topology, lots of things start to break when nodes
are not in the whitelist.
In our special case we have 1 central discovery end point, if we had
n, the problem would multiply by n.
>
> And why would a service API help mitigate that issue ?
It would give those that are able to see the deployment infrastructure
that they are dealing with, the opportunity to address this issue.
> Considering that service would have to be reconfigured, too, and your code sample belows uses a static reference ?
AFAIK it uses an optional reference, satisfied by a implementation
outside Sling. The snippet was written from memory directly into the
email to communicate what I was trying to discuss. Its is not real
code.
>
> Also, noted in another thread, this would require export of the API from the impl bundle, which I am not very happy with...
>
> Finally -- and this is purely subjective -- I am not very happy with the name WhiteListProvider and then WhiteListProvider.isWhiteListed... Also would there be other needs for hooking into this servlet ?
>
I have been working on an implementation most of the day that is
nothing like what was first discussed, I will commit in a moment. Much
better to share real code with unit test coverage.
Best Regards
Ian
> Regards
> Felix
>
>
>
>>
>> Would it be possible to have a API service that is consulted if
>> present to check if the request is allowed. For those that want to use
>> the service they would configure the whitelist to reject everything
>> while the service was not present so avoid startup issues.
>>
>> eg
>>
>> + @Reference(cardinality=ReferenceCardinality.OPTIONAL_UNARY)
>> + private WhiteListProvider whiteListProvider;
>>
>> /** Checks if the provided request's remote server is whitelisted **/
>>
>> private boolean isWhitelisted(final SlingHttpServletRequest request) {
>> + if (whiteListProvider != null) {
>> + whiteListProvider.isWhitelisted(request);
>> + }
>> if (whitelist.contains(request.getRemoteAddr())) {
>>
>> return true;
>>
>> } else if (whitelist.contains(request.getRemoteHost())) {
>>
>> return true;
>>
>> }
>>
>> logger.info("isWhitelisted: rejecting " + request.getRemoteAddr()
>>
>> + ", " + request.getRemoteHost());
>>
>> return false;
>>
>> }
>>
>>
>> and in the API, presumably discovery api.
>>
>> public interface WhiteListProvider {
>>
>> private boolean isWhitelisted(HttpServletRequest request);
>>
>> }
>>
>>
>> Best Regards
>> Ian
>
Re: TopologyConnectorServlet
Posted by Felix Meschberger <fm...@adobe.com>.
Hi
Am 08.10.2013 um 10:40 schrieb Ian Boston:
> Hi,
> The whitelist configuration in this servlet is causing some problems
> where the contents of the whitelist is potentially large and changing,
> since it requires constant re-configuration.
What is the problem with that ?
And why would a service API help mitigate that issue ? Considering that service would have to be reconfigured, too, and your code sample belows uses a static reference ?
Also, noted in another thread, this would require export of the API from the impl bundle, which I am not very happy with...
Finally -- and this is purely subjective -- I am not very happy with the name WhiteListProvider and then WhiteListProvider.isWhiteListed... Also would there be other needs for hooking into this servlet ?
Regards
Felix
>
> Would it be possible to have a API service that is consulted if
> present to check if the request is allowed. For those that want to use
> the service they would configure the whitelist to reject everything
> while the service was not present so avoid startup issues.
>
> eg
>
> + @Reference(cardinality=ReferenceCardinality.OPTIONAL_UNARY)
> + private WhiteListProvider whiteListProvider;
>
> /** Checks if the provided request's remote server is whitelisted **/
>
> private boolean isWhitelisted(final SlingHttpServletRequest request) {
> + if (whiteListProvider != null) {
> + whiteListProvider.isWhitelisted(request);
> + }
> if (whitelist.contains(request.getRemoteAddr())) {
>
> return true;
>
> } else if (whitelist.contains(request.getRemoteHost())) {
>
> return true;
>
> }
>
> logger.info("isWhitelisted: rejecting " + request.getRemoteAddr()
>
> + ", " + request.getRemoteHost());
>
> return false;
>
> }
>
>
> and in the API, presumably discovery api.
>
> public interface WhiteListProvider {
>
> private boolean isWhitelisted(HttpServletRequest request);
>
> }
>
>
> Best Regards
> Ian
Re: TopologyConnectorServlet
Posted by Stefan Egli <eg...@adobe.com>.
On 10/9/13 10:34 AM, "Ian Boston" <ie...@tfd.co.uk> wrote:
>On 9 October 2013 09:16, Stefan Egli <eg...@adobe.com> wrote:
>> Hi Ian,
>>
>> +1
>>
>> I like the idea of making an API out of this. One thing to remember is
>> that this API would be part of discovery.impl rather than discovery.api
>>-
>> since the discovery.api does not make any assumptions on how
>> instances/clusters discover each other. Nevertheless I think it's a good
>> idea.
>
>Ok, thats probably easier if its Ok for the Discovery Impl to export a
>package.
>
>I forgot 1 thing, there will need to be support for creating the
>aspects of the request that make it trusted, as well as white listing
>it.
I added a suggestion for the API to SLING-3154 - do you see additional
properties (to the plain request object) needed to decide if a request can
be trusted?
>
>
>>
>> Additionally, I wonder if we could also include the possibility of
>> encrypting/decrypting the payload that discovery.impl sends around via
>>the
>> topology connectors? It could be a separate service - eg
>> DiscoveryPayloadHandler - and provide methods to encrypt and
>> decrypt/verify.
>
>That would be a larger patch, but would also work. If its going to
>cover all bases, does it need to be outside the Discovery impl ?
I see it as an optional thing, surely. But was thinking it would be a good
idea to make the API also support that case.
Cheers,
Stefan
>
>I've started a jira for this [1]
>
>Best Regards
>Ian
>
>1 https://issues.apache.org/jira/browse/SLING-3154
>
>
>
>>
>> Cheers,
>> Stefan
>>
>> On 10/8/13 7:40 PM, "Ian Boston" <ie...@tfd.co.uk> wrote:
>>
>>>Hi,
>>>The whitelist configuration in this servlet is causing some problems
>>>where the contents of the whitelist is potentially large and changing,
>>>since it requires constant re-configuration.
>>>
>>>Would it be possible to have a API service that is consulted if
>>>present to check if the request is allowed. For those that want to use
>>>the service they would configure the whitelist to reject everything
>>>while the service was not present so avoid startup issues.
>>>
>>>eg
>>>
>>>+ @Reference(cardinality=ReferenceCardinality.OPTIONAL_UNARY)
>>>+ private WhiteListProvider whiteListProvider;
>>>
>>> /** Checks if the provided request's remote server is whitelisted
>>>**/
>>>
>>> private boolean isWhitelisted(final SlingHttpServletRequest
>>>request) {
>>>+ if (whiteListProvider != null) {
>>>+ whiteListProvider.isWhitelisted(request);
>>>+ }
>>> if (whitelist.contains(request.getRemoteAddr())) {
>>>
>>> return true;
>>>
>>> } else if (whitelist.contains(request.getRemoteHost())) {
>>>
>>> return true;
>>>
>>> }
>>>
>>> logger.info("isWhitelisted: rejecting " +
>>>request.getRemoteAddr()
>>>
>>> + ", " + request.getRemoteHost());
>>>
>>> return false;
>>>
>>> }
>>>
>>>
>>>and in the API, presumably discovery api.
>>>
>>>public interface WhiteListProvider {
>>>
>>> private boolean isWhitelisted(HttpServletRequest request);
>>>
>>>}
>>>
>>>
>>>Best Regards
>>>Ian
>>
Re: TopologyConnectorServlet
Posted by Ian Boston <ie...@tfd.co.uk>.
On 9 October 2013 09:16, Stefan Egli <eg...@adobe.com> wrote:
> Hi Ian,
>
> +1
>
> I like the idea of making an API out of this. One thing to remember is
> that this API would be part of discovery.impl rather than discovery.api -
> since the discovery.api does not make any assumptions on how
> instances/clusters discover each other. Nevertheless I think it's a good
> idea.
Ok, thats probably easier if its Ok for the Discovery Impl to export a package.
I forgot 1 thing, there will need to be support for creating the
aspects of the request that make it trusted, as well as white listing
it.
>
> Additionally, I wonder if we could also include the possibility of
> encrypting/decrypting the payload that discovery.impl sends around via the
> topology connectors? It could be a separate service - eg
> DiscoveryPayloadHandler - and provide methods to encrypt and
> decrypt/verify.
That would be a larger patch, but would also work. If its going to
cover all bases, does it need to be outside the Discovery impl ?
I've started a jira for this [1]
Best Regards
Ian
1 https://issues.apache.org/jira/browse/SLING-3154
>
> Cheers,
> Stefan
>
> On 10/8/13 7:40 PM, "Ian Boston" <ie...@tfd.co.uk> wrote:
>
>>Hi,
>>The whitelist configuration in this servlet is causing some problems
>>where the contents of the whitelist is potentially large and changing,
>>since it requires constant re-configuration.
>>
>>Would it be possible to have a API service that is consulted if
>>present to check if the request is allowed. For those that want to use
>>the service they would configure the whitelist to reject everything
>>while the service was not present so avoid startup issues.
>>
>>eg
>>
>>+ @Reference(cardinality=ReferenceCardinality.OPTIONAL_UNARY)
>>+ private WhiteListProvider whiteListProvider;
>>
>> /** Checks if the provided request's remote server is whitelisted **/
>>
>> private boolean isWhitelisted(final SlingHttpServletRequest request) {
>>+ if (whiteListProvider != null) {
>>+ whiteListProvider.isWhitelisted(request);
>>+ }
>> if (whitelist.contains(request.getRemoteAddr())) {
>>
>> return true;
>>
>> } else if (whitelist.contains(request.getRemoteHost())) {
>>
>> return true;
>>
>> }
>>
>> logger.info("isWhitelisted: rejecting " + request.getRemoteAddr()
>>
>> + ", " + request.getRemoteHost());
>>
>> return false;
>>
>> }
>>
>>
>>and in the API, presumably discovery api.
>>
>>public interface WhiteListProvider {
>>
>> private boolean isWhitelisted(HttpServletRequest request);
>>
>>}
>>
>>
>>Best Regards
>>Ian
>
Re: TopologyConnectorServlet
Posted by Stefan Egli <eg...@adobe.com>.
Hi Ian,
+1
I like the idea of making an API out of this. One thing to remember is
that this API would be part of discovery.impl rather than discovery.api -
since the discovery.api does not make any assumptions on how
instances/clusters discover each other. Nevertheless I think it's a good
idea.
Additionally, I wonder if we could also include the possibility of
encrypting/decrypting the payload that discovery.impl sends around via the
topology connectors? It could be a separate service - eg
DiscoveryPayloadHandler - and provide methods to encrypt and
decrypt/verify.
Cheers,
Stefan
On 10/8/13 7:40 PM, "Ian Boston" <ie...@tfd.co.uk> wrote:
>Hi,
>The whitelist configuration in this servlet is causing some problems
>where the contents of the whitelist is potentially large and changing,
>since it requires constant re-configuration.
>
>Would it be possible to have a API service that is consulted if
>present to check if the request is allowed. For those that want to use
>the service they would configure the whitelist to reject everything
>while the service was not present so avoid startup issues.
>
>eg
>
>+ @Reference(cardinality=ReferenceCardinality.OPTIONAL_UNARY)
>+ private WhiteListProvider whiteListProvider;
>
> /** Checks if the provided request's remote server is whitelisted **/
>
> private boolean isWhitelisted(final SlingHttpServletRequest request) {
>+ if (whiteListProvider != null) {
>+ whiteListProvider.isWhitelisted(request);
>+ }
> if (whitelist.contains(request.getRemoteAddr())) {
>
> return true;
>
> } else if (whitelist.contains(request.getRemoteHost())) {
>
> return true;
>
> }
>
> logger.info("isWhitelisted: rejecting " + request.getRemoteAddr()
>
> + ", " + request.getRemoteHost());
>
> return false;
>
> }
>
>
>and in the API, presumably discovery api.
>
>public interface WhiteListProvider {
>
> private boolean isWhitelisted(HttpServletRequest request);
>
>}
>
>
>Best Regards
>Ian