You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@knox.apache.org by Dilli Arumugam <da...@hortonworks.com> on 2013/10/25 16:27:13 UTC
some design thoughts on knox - simplify integrating new service,
configuration, dependency injection
Have been trying out a few ideas with Knox and managed to get Knox running
with
1. WEB-INF/gateway.xml is completely removed
2. WEB-INF/web.xml declares only shiro filter and a defautl servlet
2. all filters are defined and injected using WEB-INF/shiro.ini
To me that looks much simpler to comprehend and administer.
Agreed, this could be subjective.
Hence, seeking comments from community.
Pasting web.xml and shiro.ini inline.
web.xml
--------------
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<web-app xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="
http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="
http://java.sun.com/xml/ns/javaee
http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd">
<listener>
<listener-class>org.apache.shiro.web.env.EnvironmentLoaderListener</listener-class>
</listener>
<listener>
<listener-class>org.apache.hadoop.gateway.services.GatewayServicesContextListener</listener-class>
</listener>
<listener>
<listener-class>org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletContextListener</listener-class>
</listener>
<context-param>
<param-name>rewriteDescriptorLocation</param-name>
<param-value>/WEB-INF/rewrite.xml</param-value>
</context-param>
<filter>
<filter-name>ShiroFilter</filter-name>
<filter-class>org.apache.shiro.web.servlet.ShiroFilter</filter-class>
</filter>
<filter-mapping>
<filter-name>ShiroFilter</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>
<session-config>
<session-timeout>30</session-timeout>
</session-config>
<servlet>
<servlet-name>errorservlet</servlet-name>
<servlet-class>org.apache.hadoop.gateway.filter.KnoxErrorServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>errorservlet</servlet-name>
<url-pattern>/*</url-pattern>
</servlet-mapping>
</web-app>
shiro.ini
-------------
[main]
# define ldapRealm
ldapRealm=org.apache.shiro.realm.ldap.JndiLdapRealm
ldapRealm.contextFactory.authenticationMechanism=simple
ldapRealm.contextFactory.url=ldap://localhost:33389
ldapRealm.userDnTemplate=uid={0},ou=people,dc=hadoop,dc=apache,dc=org
# define filter: knoxResponseCookieFilter
knoxResponseCookieFilter =
org.apache.hadoop.gateway.filter.ResponseCookieFilter
knoxResponseCookieFilter.enabled = true
knoxResponseCookieFilter.filterHeaders = rememberMe, hadoop.auth.cookie
# define filter: knoxPrincipalMapper
knoxPrincipalMapper = org.apache.hadoop.gateway.filter.KnoxPrincipalMapper
knoxPrincipalMapper.enabled = true
knoxPrincipalMapper.userToUserMap = bob:guest, jon:bob
knoxPrincipalMapper.userToGroupMap = *:users, bob:admin
# define filter: knoxIPTracker
knoxIPTracker =org.apache.hadoop.gateway.filter.KnoxIPTracker
knoxIPTracker.enabled = true
# define filter: knoxAclAuthzFilter
knoxAclAuthzFilter = org.apache.hadoop.gateway.filter.KnoxAclAuthzFilter
knoxAclAuthzFilter.enabled = true
knoxAclAuthzFilter.globalGroupAclMode = OR
knoxAclAuthzFilter.serviceGroupAclModeMap = /webhdfs/:OR, /templeton/:OR
knoxAclAuthzFilter.serviceAclMap = /webhdfs/:user1 user2; users admin;
127.*.*.*, /templeton/:user11 user12; users admin
# define filter: javaSubjectMapper
javaSubjectMapper =
org.apache.hadoop.gateway.filter.PostAuthenticationFilter
# define filter: knoxIdentityAsserter
knoxIdentityAsserter = org.apache.hadoop.gateway.filter.KnoxIdentityAsserter
knoxIdentityAsserter.enabled = true
# define filter: knoxUrlRewriter
knoxUrlRewriter =
org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter
knoxUrlRewriter.requestUrlMap =
/webhdfs/v1/:WEBHDFS/webhdfs/inbound/namenode/root,
/webhdfs/v1/[^~]+:WEBHDFS/webhdfs/inbound/namenode/file,
/webhdfs/v1/~:WEBHDFS/webhdfs/inbound/namenode/home,
/webhdfs/v1/~/.*:WEBHDFS/webhdfs/inbound/namenode/home/file,
/webhdfs/data/v1/.+:WEBHDFS/webhdfs/inbound/datanode
knoxUrlRewriter.requestBodyMap = /oozie/:OOZIE/oozie/configuration,
/oozie/v1/.*:OOZIE/oozie/configuration,
/oozie/v2/.*:OOZIE/oozie/configuration
knoxUrlRewriter.responseHeadersMap =
/webhdfs/v1/[^~]*:WEBHDFS/webhdfs/outbound/namenode/headers,
/webhdfs/v1/~/:WEBHDFS/webhdfs/outbound/namenode/headers,
/hbase/:WEBHBASE/webhbase/headers/outbound,
/hbase/[~/]*:WEBHBASE/webhbase/headers/outbound,
/hbase/status/cluster:WEBHBASE/webhbase/status/outbound,
/hbase/[^/]*/regions:WEBHBASE/webhbase/regions/outbound
# define filter: knoxHttpDispatcher
knoxHttpDispatcher = org.apache.hadoop.gateway.dispatch.HttpClientDispatch
knoxHttpDispatcher.replayBufferSizeMap = webhdfs:4, templeton:8, oozie:4
[urls]
# you could choose to have a different chain of filter for different url
patterns
# so far Knox did not need it
/** = knoxResponseCookieFilter, authcBasic, knoxPrincipalMapper,
knoxIPTracker, knoxAclAuthzFilter, javaSubjectMapper, knoxIdentityAsserter,
knoxUrlRewriter, knoxHttpDispatcher
# end of shiro.ini
I think we can rename shiro.ini as knox.ini to make it explicit this is
more about knox configuration than shiro library configuration.
We are using shiro config file mechanism as simple, lightweight depenency
injection.
This does not really tie us to using only Shiro authentication mechanism or
authorization mechanisms.
We have the choice of writing all our authentication or authorization in
our own servlet filters or leverage filters from Shiro library, Realms from
Shiro library or write your own Shiro filters or Shiro realms.
In most of the cases, when we want to integrate with a new Hadoop back end
service, all that we have to do is specify the path to a file having
rewrite rules for the service. Rest of the things in knox.ini(=shiro.ini)
would remain same.
Please review and comment.
Thanks
Dilli
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: some design thoughts on knox - simplify integrating new service,
configuration, dependency injection
Posted by larry mccay <la...@gmail.com>.
It seems to me that we should have very simple config for the endpoints and
other deployment specific config added to a shiro.ini that has all the
boilerplate stuff in it. Then you can make the rewrite rules stay in there
as well.
Which, btw, is the idea behind topology.xml - it is a simple deployment
specific descriptor that gets ingested and *compiled* into effective
deployment configuration. Today that is gateway.xml and what you are
proposing is shiro.ini.
On Fri, Oct 25, 2013 at 12:25 PM, Dilli Arumugam
<da...@hortonworks.com>wrote:
> Larry,
>
> I would go with specifying the end points of hadoop services in
> knox.ini(=shiro.ini) and leave the rewrite rules in its own file.
>
> Rewrite rules are not going to be cluster specific and admins typically
> would not have to tinker with them.
>
> Thanks
> Dilli
>
>
> On Fri, Oct 25, 2013 at 9:03 AM, Dilli Arumugam
> <da...@hortonworks.com>wrote:
>
> > Thanks Larry for the review and comments.
> >
> > The proposed knox.ini is meant to be edited by Knox system administrator
> > until we get to have a admin console. This is not meant to be edited by
> end
> > user.
> >
> > To integrate additional back end hadoop service, following pieces of
> > information have to come from pointers from knox.ini to external
> > configuration files.
> >
> > 1. end point of the new hadoop servie back end
> > 2. rewriting rules for the new service
> >
> > These can be specified in one file or split between 2 files with one of
> > them being topology file or its equivalent.
> > I have not shown all those details in the current shiro.ini.
> > But, you would get the idea.
> >
> > Dilli
> >
> >
> > On Fri, Oct 25, 2013 at 8:50 AM, larry mccay <lm...@apache.org> wrote:
> >
> >> Hi Dilli -
> >>
> >> Good work here!
> >>
> >> Question:
> >> 1. Is this complicated ini hand edited by the enduser in order to
> deploy a
> >> given topology or have you changed the shiro contributor to write this
> out
> >> from a much more understandable config within the topology?
> >>
> >> 2. Where are the services configured for a given topology?
> >> 3. Did you rename all the existing filters to start with knox?
> >>
> >> A couple observations:
> >>
> >> 1. removing the use of gateway.xml means that we *require* Shiro for our
> >> servlet filter chaining
> >> 2. While comparing this to the current format of gateway.xml I can agree
> >> that this is easier to understand - however, I don't actually find ini
> >> format simpler to comprehend and administer than XML or JSON. In fact,
> >> comparing this to topology.xml which is all you really need to do for
> >> admin, I think the topology file is very easy to understand and
> >> administer.
> >> 3. moving the rewrite rules out to not require coding is good but should
> >> probably not require that they be in this format and file
> >> 4. Someone that wants to leverage their own or another third party
> >> solution
> >> is stuck squeezing it into Shiro
> >> 5. This would violate a principle of mine of "Is-a" vs "Has-a" - Knox is
> >> not a Shiro provider it can however *have* a Shiro provider
> >> 6. End users should never have to see rewrite rules and maybe a lot of
> >> what
> >> is in that ini. Principal mapping, authz policy, etc will eventually
> come
> >> from Zookeeper which should remove their need to be in this file for
> those
> >> things. Maybe we can even get the LDAP config separately from there as
> >> well
> >> - though I still think it is best provided in a topology file with a
> much
> >> smaller config requirements and get pushed into shiro.ini. Then the user
> >> may never need to see - or at least open - this file.
> >>
> >> I continue to believe that we are best served by enabling a deployment
> >> that
> >> looks like this and encouraging the use of it as the preferred solution.
> >> However, we should continue to support the use of arbitrary providers
> >> through the use of gateway.xml. In most cases, gateway.xml can be
> >> leveraged
> >> to only include the Shiro provider. In addition, all knowledge of Shiro
> >> specifics should remain inside of that provider. If it leaks out then we
> >> are violating the separation of Is-a vs Has-a again.
> >>
> >> This also allows us the flexibility to add something before or after
> that
> >> provider for things that we find are not easily or appropriately done
> >> inside the security provider.
> >>
> >> "I think we can rename shiro.ini as knox.ini to make it explicit this is
> >> more about knox configuration than shiro library configuration."
> >>
> >> While I can understand the thinking of renaming the shiro.ini file to
> >> knox.ini, also violates the Is-a vs Has-a principle for me. We are not a
> >> shiro security provider extension. We are a REST API Gateway for Hadoop
> >> that has security providers - one of which is Shiro.
> >>
> >> "We are using shiro config file mechanism as simple, lightweight
> depenency
> >> injection."
> >>
> >> Where is there dependency injection here? If you are referring to the
> >> configuration of the filter chain that is not really injection though it
> >> is
> >> an admittedly simpler mechanism than the current state of the
> contributors
> >> and our deployment machinery - which could be refactored and improved.
> >>
> >> "This does not really tie us to using only Shiro authentication
> mechanism
> >> or
> >> authorization mechanisms.
> >>
> >> We have the choice of writing all our authentication or authorization in
> >> our own servlet filters or leverage filters from Shiro library, Realms
> >> from
> >> Shiro library or write your own Shiro filters or Shiro realms."
> >>
> >> The only change that this proposal makes to our flexibility is the
> >> inability to take Shiro out of the picture.
> >> We could have leveraged Shiro to do all these things with the current
> >> design as well.
> >> BUT, we have control over our own interception channel in our chains
> >> today.
> >> This proposal takes a specific *security provider's* proprietary filter
> >> chain and uses it as the only interceptor channel for the entire server.
> >> Just because they also had a need for one - for their security work -
> >> doesn't mean that we shouldn't have our own. The tail should not wag the
> >> dog. Shiro is not an application or server framework it is a security
> >> provider.
> >>
> >> "In most of the cases, when we want to integrate with a new Hadoop back
> >> end
> >> service, all that we have to do is specify the path to a file having
> >> rewrite rules for the service. Rest of the things in
> knox.ini(=shiro.ini)
> >> would remain same."
> >>
> >> This isn't really very clear to me. Do we still have services in a
> >> topology.xml file?
> >> I don't see where you specify the url for the backend services - so I
> >> assume there is still a topology file.
> >> How do we know where to dispatch the requests to?
> >>
> >> I do agree that this is a very good step forward in defining a preferred
> >> security provider model for Knox and that we should continue down this
> >> road. We just need to do so carefully. If, over time, we have found no
> >> reason to use the existing solution that we can consider removing it
> >> entirely.
> >>
> >> Thanks for all this work and insight, Dilli!
> >>
> >> --larry
> >>
> >>
> >> On Fri, Oct 25, 2013 at 10:27 AM, Dilli Arumugam
> >> <da...@hortonworks.com>wrote:
> >>
> >> > Have been trying out a few ideas with Knox and managed to get Knox
> >> running
> >> > with
> >> >
> >> > 1. WEB-INF/gateway.xml is completely removed
> >> > 2. WEB-INF/web.xml declares only shiro filter and a defautl servlet
> >> > 2. all filters are defined and injected using WEB-INF/shiro.ini
> >> >
> >> > To me that looks much simpler to comprehend and administer.
> >> > Agreed, this could be subjective.
> >> > Hence, seeking comments from community.
> >> >
> >> > Pasting web.xml and shiro.ini inline.
> >> >
> >> > web.xml
> >> > --------------
> >> >
> >> > <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> >> > <web-app xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="
> >> > http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="
> >> > http://java.sun.com/xml/ns/javaee
> >> > http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd">
> >> >
> >> > <listener>
> >> >
> >> >
> >> >
> >>
> <listener-class>org.apache.shiro.web.env.EnvironmentLoaderListener</listener-class>
> >> > </listener>
> >> >
> >> > <listener>
> >> >
> >> >
> >> >
> >>
> <listener-class>org.apache.hadoop.gateway.services.GatewayServicesContextListener</listener-class>
> >> > </listener>
> >> >
> >> > <listener>
> >> >
> >> >
> >> >
> >>
> <listener-class>org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletContextListener</listener-class>
> >> > </listener>
> >> >
> >> > <context-param>
> >> > <param-name>rewriteDescriptorLocation</param-name>
> >> > <param-value>/WEB-INF/rewrite.xml</param-value>
> >> > </context-param>
> >> >
> >> > <filter>
> >> > <filter-name>ShiroFilter</filter-name>
> >> >
> >> <filter-class>org.apache.shiro.web.servlet.ShiroFilter</filter-class>
> >> > </filter>
> >> >
> >> > <filter-mapping>
> >> > <filter-name>ShiroFilter</filter-name>
> >> > <url-pattern>/*</url-pattern>
> >> > </filter-mapping>
> >> >
> >> > <session-config>
> >> > <session-timeout>30</session-timeout>
> >> > </session-config>
> >> >
> >> > <servlet>
> >> > <servlet-name>errorservlet</servlet-name>
> >> >
> >> >
> >> >
> >>
> <servlet-class>org.apache.hadoop.gateway.filter.KnoxErrorServlet</servlet-class>
> >> > </servlet>
> >> >
> >> > <servlet-mapping>
> >> > <servlet-name>errorservlet</servlet-name>
> >> > <url-pattern>/*</url-pattern>
> >> > </servlet-mapping>
> >> >
> >> > </web-app>
> >> >
> >> >
> >> > shiro.ini
> >> > -------------
> >> >
> >> > [main]
> >> >
> >> > # define ldapRealm
> >> > ldapRealm=org.apache.shiro.realm.ldap.JndiLdapRealm
> >> > ldapRealm.contextFactory.authenticationMechanism=simple
> >> > ldapRealm.contextFactory.url=ldap://localhost:33389
> >> > ldapRealm.userDnTemplate=uid={0},ou=people,dc=hadoop,dc=apache,dc=org
> >> >
> >> >
> >> > # define filter: knoxResponseCookieFilter
> >> > knoxResponseCookieFilter =
> >> > org.apache.hadoop.gateway.filter.ResponseCookieFilter
> >> > knoxResponseCookieFilter.enabled = true
> >> > knoxResponseCookieFilter.filterHeaders = rememberMe,
> hadoop.auth.cookie
> >> >
> >> > # define filter: knoxPrincipalMapper
> >> > knoxPrincipalMapper =
> >> org.apache.hadoop.gateway.filter.KnoxPrincipalMapper
> >> > knoxPrincipalMapper.enabled = true
> >> > knoxPrincipalMapper.userToUserMap = bob:guest, jon:bob
> >> > knoxPrincipalMapper.userToGroupMap = *:users, bob:admin
> >> >
> >> > # define filter: knoxIPTracker
> >> > knoxIPTracker =org.apache.hadoop.gateway.filter.KnoxIPTracker
> >> > knoxIPTracker.enabled = true
> >> >
> >> > # define filter: knoxAclAuthzFilter
> >> > knoxAclAuthzFilter =
> org.apache.hadoop.gateway.filter.KnoxAclAuthzFilter
> >> > knoxAclAuthzFilter.enabled = true
> >> > knoxAclAuthzFilter.globalGroupAclMode = OR
> >> > knoxAclAuthzFilter.serviceGroupAclModeMap = /webhdfs/:OR,
> /templeton/:OR
> >> > knoxAclAuthzFilter.serviceAclMap = /webhdfs/:user1 user2; users admin;
> >> > 127.*.*.*, /templeton/:user11 user12; users admin
> >> >
> >> > # define filter: javaSubjectMapper
> >> > javaSubjectMapper =
> >> > org.apache.hadoop.gateway.filter.PostAuthenticationFilter
> >> >
> >> > # define filter: knoxIdentityAsserter
> >> > knoxIdentityAsserter =
> >> > org.apache.hadoop.gateway.filter.KnoxIdentityAsserter
> >> > knoxIdentityAsserter.enabled = true
> >> >
> >> > # define filter: knoxUrlRewriter
> >> > knoxUrlRewriter =
> >> > org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter
> >> >
> >> > knoxUrlRewriter.requestUrlMap =
> >> > /webhdfs/v1/:WEBHDFS/webhdfs/inbound/namenode/root,
> >> > /webhdfs/v1/[^~]+:WEBHDFS/webhdfs/inbound/namenode/file,
> >> > /webhdfs/v1/~:WEBHDFS/webhdfs/inbound/namenode/home,
> >> > /webhdfs/v1/~/.*:WEBHDFS/webhdfs/inbound/namenode/home/file,
> >> > /webhdfs/data/v1/.+:WEBHDFS/webhdfs/inbound/datanode
> >> >
> >> > knoxUrlRewriter.requestBodyMap = /oozie/:OOZIE/oozie/configuration,
> >> > /oozie/v1/.*:OOZIE/oozie/configuration,
> >> > /oozie/v2/.*:OOZIE/oozie/configuration
> >> >
> >> > knoxUrlRewriter.responseHeadersMap =
> >> > /webhdfs/v1/[^~]*:WEBHDFS/webhdfs/outbound/namenode/headers,
> >> > /webhdfs/v1/~/:WEBHDFS/webhdfs/outbound/namenode/headers,
> >> > /hbase/:WEBHBASE/webhbase/headers/outbound,
> >> > /hbase/[~/]*:WEBHBASE/webhbase/headers/outbound,
> >> > /hbase/status/cluster:WEBHBASE/webhbase/status/outbound,
> >> > /hbase/[^/]*/regions:WEBHBASE/webhbase/regions/outbound
> >> >
> >> > # define filter: knoxHttpDispatcher
> >> > knoxHttpDispatcher =
> >> org.apache.hadoop.gateway.dispatch.HttpClientDispatch
> >> > knoxHttpDispatcher.replayBufferSizeMap = webhdfs:4, templeton:8,
> oozie:4
> >> >
> >> > [urls]
> >> > # you could choose to have a different chain of filter for different
> url
> >> > patterns
> >> > # so far Knox did not need it
> >> > /** = knoxResponseCookieFilter, authcBasic, knoxPrincipalMapper,
> >> > knoxIPTracker, knoxAclAuthzFilter, javaSubjectMapper,
> >> knoxIdentityAsserter,
> >> > knoxUrlRewriter, knoxHttpDispatcher
> >> >
> >> > # end of shiro.ini
> >> >
> >> > I think we can rename shiro.ini as knox.ini to make it explicit this
> is
> >> > more about knox configuration than shiro library configuration.
> >> >
> >> >
> >> > We are using shiro config file mechanism as simple, lightweight
> >> depenency
> >> > injection.
> >> >
> >> > This does not really tie us to using only Shiro authentication
> >> mechanism or
> >> > authorization mechanisms.
> >> >
> >> > We have the choice of writing all our authentication or authorization
> in
> >> > our own servlet filters or leverage filters from Shiro library, Realms
> >> from
> >> > Shiro library or write your own Shiro filters or Shiro realms.
> >> >
> >> > In most of the cases, when we want to integrate with a new Hadoop back
> >> end
> >> > service, all that we have to do is specify the path to a file having
> >> > rewrite rules for the service. Rest of the things in
> >> knox.ini(=shiro.ini)
> >> > would remain same.
> >> >
> >> > Please review and comment.
> >> >
> >> >
> >> > Thanks
> >> > Dilli
> >> >
> >> > --
> >> > CONFIDENTIALITY NOTICE
> >> > NOTICE: This message is intended for the use of the individual or
> >> entity to
> >> > which it is addressed and may contain information that is
> confidential,
> >> > privileged and exempt from disclosure under applicable law. If the
> >> reader
> >> > of this message is not the intended recipient, you are hereby notified
> >> that
> >> > any printing, copying, dissemination, distribution, disclosure or
> >> > forwarding of this communication is strictly prohibited. If you have
> >> > received this communication in error, please contact the sender
> >> immediately
> >> > and delete it from your system. Thank You.
> >> >
> >>
> >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: some design thoughts on knox - simplify integrating new service,
configuration, dependency injection
Posted by Dilli Arumugam <da...@hortonworks.com>.
Larry,
I would go with specifying the end points of hadoop services in
knox.ini(=shiro.ini) and leave the rewrite rules in its own file.
Rewrite rules are not going to be cluster specific and admins typically
would not have to tinker with them.
Thanks
Dilli
On Fri, Oct 25, 2013 at 9:03 AM, Dilli Arumugam
<da...@hortonworks.com>wrote:
> Thanks Larry for the review and comments.
>
> The proposed knox.ini is meant to be edited by Knox system administrator
> until we get to have a admin console. This is not meant to be edited by end
> user.
>
> To integrate additional back end hadoop service, following pieces of
> information have to come from pointers from knox.ini to external
> configuration files.
>
> 1. end point of the new hadoop servie back end
> 2. rewriting rules for the new service
>
> These can be specified in one file or split between 2 files with one of
> them being topology file or its equivalent.
> I have not shown all those details in the current shiro.ini.
> But, you would get the idea.
>
> Dilli
>
>
> On Fri, Oct 25, 2013 at 8:50 AM, larry mccay <lm...@apache.org> wrote:
>
>> Hi Dilli -
>>
>> Good work here!
>>
>> Question:
>> 1. Is this complicated ini hand edited by the enduser in order to deploy a
>> given topology or have you changed the shiro contributor to write this out
>> from a much more understandable config within the topology?
>>
>> 2. Where are the services configured for a given topology?
>> 3. Did you rename all the existing filters to start with knox?
>>
>> A couple observations:
>>
>> 1. removing the use of gateway.xml means that we *require* Shiro for our
>> servlet filter chaining
>> 2. While comparing this to the current format of gateway.xml I can agree
>> that this is easier to understand - however, I don't actually find ini
>> format simpler to comprehend and administer than XML or JSON. In fact,
>> comparing this to topology.xml which is all you really need to do for
>> admin, I think the topology file is very easy to understand and
>> administer.
>> 3. moving the rewrite rules out to not require coding is good but should
>> probably not require that they be in this format and file
>> 4. Someone that wants to leverage their own or another third party
>> solution
>> is stuck squeezing it into Shiro
>> 5. This would violate a principle of mine of "Is-a" vs "Has-a" - Knox is
>> not a Shiro provider it can however *have* a Shiro provider
>> 6. End users should never have to see rewrite rules and maybe a lot of
>> what
>> is in that ini. Principal mapping, authz policy, etc will eventually come
>> from Zookeeper which should remove their need to be in this file for those
>> things. Maybe we can even get the LDAP config separately from there as
>> well
>> - though I still think it is best provided in a topology file with a much
>> smaller config requirements and get pushed into shiro.ini. Then the user
>> may never need to see - or at least open - this file.
>>
>> I continue to believe that we are best served by enabling a deployment
>> that
>> looks like this and encouraging the use of it as the preferred solution.
>> However, we should continue to support the use of arbitrary providers
>> through the use of gateway.xml. In most cases, gateway.xml can be
>> leveraged
>> to only include the Shiro provider. In addition, all knowledge of Shiro
>> specifics should remain inside of that provider. If it leaks out then we
>> are violating the separation of Is-a vs Has-a again.
>>
>> This also allows us the flexibility to add something before or after that
>> provider for things that we find are not easily or appropriately done
>> inside the security provider.
>>
>> "I think we can rename shiro.ini as knox.ini to make it explicit this is
>> more about knox configuration than shiro library configuration."
>>
>> While I can understand the thinking of renaming the shiro.ini file to
>> knox.ini, also violates the Is-a vs Has-a principle for me. We are not a
>> shiro security provider extension. We are a REST API Gateway for Hadoop
>> that has security providers - one of which is Shiro.
>>
>> "We are using shiro config file mechanism as simple, lightweight depenency
>> injection."
>>
>> Where is there dependency injection here? If you are referring to the
>> configuration of the filter chain that is not really injection though it
>> is
>> an admittedly simpler mechanism than the current state of the contributors
>> and our deployment machinery - which could be refactored and improved.
>>
>> "This does not really tie us to using only Shiro authentication mechanism
>> or
>> authorization mechanisms.
>>
>> We have the choice of writing all our authentication or authorization in
>> our own servlet filters or leverage filters from Shiro library, Realms
>> from
>> Shiro library or write your own Shiro filters or Shiro realms."
>>
>> The only change that this proposal makes to our flexibility is the
>> inability to take Shiro out of the picture.
>> We could have leveraged Shiro to do all these things with the current
>> design as well.
>> BUT, we have control over our own interception channel in our chains
>> today.
>> This proposal takes a specific *security provider's* proprietary filter
>> chain and uses it as the only interceptor channel for the entire server.
>> Just because they also had a need for one - for their security work -
>> doesn't mean that we shouldn't have our own. The tail should not wag the
>> dog. Shiro is not an application or server framework it is a security
>> provider.
>>
>> "In most of the cases, when we want to integrate with a new Hadoop back
>> end
>> service, all that we have to do is specify the path to a file having
>> rewrite rules for the service. Rest of the things in knox.ini(=shiro.ini)
>> would remain same."
>>
>> This isn't really very clear to me. Do we still have services in a
>> topology.xml file?
>> I don't see where you specify the url for the backend services - so I
>> assume there is still a topology file.
>> How do we know where to dispatch the requests to?
>>
>> I do agree that this is a very good step forward in defining a preferred
>> security provider model for Knox and that we should continue down this
>> road. We just need to do so carefully. If, over time, we have found no
>> reason to use the existing solution that we can consider removing it
>> entirely.
>>
>> Thanks for all this work and insight, Dilli!
>>
>> --larry
>>
>>
>> On Fri, Oct 25, 2013 at 10:27 AM, Dilli Arumugam
>> <da...@hortonworks.com>wrote:
>>
>> > Have been trying out a few ideas with Knox and managed to get Knox
>> running
>> > with
>> >
>> > 1. WEB-INF/gateway.xml is completely removed
>> > 2. WEB-INF/web.xml declares only shiro filter and a defautl servlet
>> > 2. all filters are defined and injected using WEB-INF/shiro.ini
>> >
>> > To me that looks much simpler to comprehend and administer.
>> > Agreed, this could be subjective.
>> > Hence, seeking comments from community.
>> >
>> > Pasting web.xml and shiro.ini inline.
>> >
>> > web.xml
>> > --------------
>> >
>> > <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
>> > <web-app xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="
>> > http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="
>> > http://java.sun.com/xml/ns/javaee
>> > http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd">
>> >
>> > <listener>
>> >
>> >
>> >
>> <listener-class>org.apache.shiro.web.env.EnvironmentLoaderListener</listener-class>
>> > </listener>
>> >
>> > <listener>
>> >
>> >
>> >
>> <listener-class>org.apache.hadoop.gateway.services.GatewayServicesContextListener</listener-class>
>> > </listener>
>> >
>> > <listener>
>> >
>> >
>> >
>> <listener-class>org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletContextListener</listener-class>
>> > </listener>
>> >
>> > <context-param>
>> > <param-name>rewriteDescriptorLocation</param-name>
>> > <param-value>/WEB-INF/rewrite.xml</param-value>
>> > </context-param>
>> >
>> > <filter>
>> > <filter-name>ShiroFilter</filter-name>
>> >
>> <filter-class>org.apache.shiro.web.servlet.ShiroFilter</filter-class>
>> > </filter>
>> >
>> > <filter-mapping>
>> > <filter-name>ShiroFilter</filter-name>
>> > <url-pattern>/*</url-pattern>
>> > </filter-mapping>
>> >
>> > <session-config>
>> > <session-timeout>30</session-timeout>
>> > </session-config>
>> >
>> > <servlet>
>> > <servlet-name>errorservlet</servlet-name>
>> >
>> >
>> >
>> <servlet-class>org.apache.hadoop.gateway.filter.KnoxErrorServlet</servlet-class>
>> > </servlet>
>> >
>> > <servlet-mapping>
>> > <servlet-name>errorservlet</servlet-name>
>> > <url-pattern>/*</url-pattern>
>> > </servlet-mapping>
>> >
>> > </web-app>
>> >
>> >
>> > shiro.ini
>> > -------------
>> >
>> > [main]
>> >
>> > # define ldapRealm
>> > ldapRealm=org.apache.shiro.realm.ldap.JndiLdapRealm
>> > ldapRealm.contextFactory.authenticationMechanism=simple
>> > ldapRealm.contextFactory.url=ldap://localhost:33389
>> > ldapRealm.userDnTemplate=uid={0},ou=people,dc=hadoop,dc=apache,dc=org
>> >
>> >
>> > # define filter: knoxResponseCookieFilter
>> > knoxResponseCookieFilter =
>> > org.apache.hadoop.gateway.filter.ResponseCookieFilter
>> > knoxResponseCookieFilter.enabled = true
>> > knoxResponseCookieFilter.filterHeaders = rememberMe, hadoop.auth.cookie
>> >
>> > # define filter: knoxPrincipalMapper
>> > knoxPrincipalMapper =
>> org.apache.hadoop.gateway.filter.KnoxPrincipalMapper
>> > knoxPrincipalMapper.enabled = true
>> > knoxPrincipalMapper.userToUserMap = bob:guest, jon:bob
>> > knoxPrincipalMapper.userToGroupMap = *:users, bob:admin
>> >
>> > # define filter: knoxIPTracker
>> > knoxIPTracker =org.apache.hadoop.gateway.filter.KnoxIPTracker
>> > knoxIPTracker.enabled = true
>> >
>> > # define filter: knoxAclAuthzFilter
>> > knoxAclAuthzFilter = org.apache.hadoop.gateway.filter.KnoxAclAuthzFilter
>> > knoxAclAuthzFilter.enabled = true
>> > knoxAclAuthzFilter.globalGroupAclMode = OR
>> > knoxAclAuthzFilter.serviceGroupAclModeMap = /webhdfs/:OR, /templeton/:OR
>> > knoxAclAuthzFilter.serviceAclMap = /webhdfs/:user1 user2; users admin;
>> > 127.*.*.*, /templeton/:user11 user12; users admin
>> >
>> > # define filter: javaSubjectMapper
>> > javaSubjectMapper =
>> > org.apache.hadoop.gateway.filter.PostAuthenticationFilter
>> >
>> > # define filter: knoxIdentityAsserter
>> > knoxIdentityAsserter =
>> > org.apache.hadoop.gateway.filter.KnoxIdentityAsserter
>> > knoxIdentityAsserter.enabled = true
>> >
>> > # define filter: knoxUrlRewriter
>> > knoxUrlRewriter =
>> > org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter
>> >
>> > knoxUrlRewriter.requestUrlMap =
>> > /webhdfs/v1/:WEBHDFS/webhdfs/inbound/namenode/root,
>> > /webhdfs/v1/[^~]+:WEBHDFS/webhdfs/inbound/namenode/file,
>> > /webhdfs/v1/~:WEBHDFS/webhdfs/inbound/namenode/home,
>> > /webhdfs/v1/~/.*:WEBHDFS/webhdfs/inbound/namenode/home/file,
>> > /webhdfs/data/v1/.+:WEBHDFS/webhdfs/inbound/datanode
>> >
>> > knoxUrlRewriter.requestBodyMap = /oozie/:OOZIE/oozie/configuration,
>> > /oozie/v1/.*:OOZIE/oozie/configuration,
>> > /oozie/v2/.*:OOZIE/oozie/configuration
>> >
>> > knoxUrlRewriter.responseHeadersMap =
>> > /webhdfs/v1/[^~]*:WEBHDFS/webhdfs/outbound/namenode/headers,
>> > /webhdfs/v1/~/:WEBHDFS/webhdfs/outbound/namenode/headers,
>> > /hbase/:WEBHBASE/webhbase/headers/outbound,
>> > /hbase/[~/]*:WEBHBASE/webhbase/headers/outbound,
>> > /hbase/status/cluster:WEBHBASE/webhbase/status/outbound,
>> > /hbase/[^/]*/regions:WEBHBASE/webhbase/regions/outbound
>> >
>> > # define filter: knoxHttpDispatcher
>> > knoxHttpDispatcher =
>> org.apache.hadoop.gateway.dispatch.HttpClientDispatch
>> > knoxHttpDispatcher.replayBufferSizeMap = webhdfs:4, templeton:8, oozie:4
>> >
>> > [urls]
>> > # you could choose to have a different chain of filter for different url
>> > patterns
>> > # so far Knox did not need it
>> > /** = knoxResponseCookieFilter, authcBasic, knoxPrincipalMapper,
>> > knoxIPTracker, knoxAclAuthzFilter, javaSubjectMapper,
>> knoxIdentityAsserter,
>> > knoxUrlRewriter, knoxHttpDispatcher
>> >
>> > # end of shiro.ini
>> >
>> > I think we can rename shiro.ini as knox.ini to make it explicit this is
>> > more about knox configuration than shiro library configuration.
>> >
>> >
>> > We are using shiro config file mechanism as simple, lightweight
>> depenency
>> > injection.
>> >
>> > This does not really tie us to using only Shiro authentication
>> mechanism or
>> > authorization mechanisms.
>> >
>> > We have the choice of writing all our authentication or authorization in
>> > our own servlet filters or leverage filters from Shiro library, Realms
>> from
>> > Shiro library or write your own Shiro filters or Shiro realms.
>> >
>> > In most of the cases, when we want to integrate with a new Hadoop back
>> end
>> > service, all that we have to do is specify the path to a file having
>> > rewrite rules for the service. Rest of the things in
>> knox.ini(=shiro.ini)
>> > would remain same.
>> >
>> > Please review and comment.
>> >
>> >
>> > Thanks
>> > Dilli
>> >
>> > --
>> > CONFIDENTIALITY NOTICE
>> > NOTICE: This message is intended for the use of the individual or
>> entity to
>> > which it is addressed and may contain information that is confidential,
>> > privileged and exempt from disclosure under applicable law. If the
>> reader
>> > of this message is not the intended recipient, you are hereby notified
>> that
>> > any printing, copying, dissemination, distribution, disclosure or
>> > forwarding of this communication is strictly prohibited. If you have
>> > received this communication in error, please contact the sender
>> immediately
>> > and delete it from your system. Thank You.
>> >
>>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: some design thoughts on knox - simplify integrating new service,
configuration, dependency injection
Posted by Dilli Arumugam <da...@hortonworks.com>.
I did not change the internals of rewriter implementation or dispatcher
implementation for the POC.
So, it should be getting these from wherever it was getting it from without
my POC change.
I need some time and help from you/Kevin to comprehend what goes on in the
internals of rewriter.
My proposal attempts to expose the filters and filter chain to
administrator.
gateway.xml hides the filters and chain from the administrator.
They have their pros and cons.
Would leave it to the choice of the community.
Thanks
Dilli
On Fri, Oct 25, 2013 at 9:36 AM, larry mccay <la...@gmail.com> wrote:
> hmmmm, so, where does the dispatcher get the endpoint info from - the
> topology file?
> That isn't even in WEB-INF.
>
>
> On Fri, Oct 25, 2013 at 12:30 PM, Dilli Arumugam
> <da...@hortonworks.com>wrote:
>
> > The POC is done starting with a knox web app already deployed.
> > Then,
> >
> > 1. remove WEB-INF/gateway.xml
> > 2. update WEB-INF/web.xml
> > 3. update WEB-INF/shiro.ini
> >
> > So, the web app was first deployed using current knox contributor
> > framework.
> > Have not spent time to simulate the equivalent of contribution
> framework.
> >
> > Thanks
> > Dilli
> >
> >
> > On Fri, Oct 25, 2013 at 9:23 AM, larry mccay <la...@gmail.com>
> > wrote:
> >
> > > On Fri, Oct 25, 2013 at 12:03 PM, Dilli Arumugam
> > > <da...@hortonworks.com>wrote:
> > >
> > > > Thanks Larry for the review and comments.
> > > >
> > > > The proposed knox.ini is meant to be edited by Knox system
> > administrator
> > > > until we get to have a admin console. This is not meant to be edited
> by
> > > end
> > > > user.
> > > >
> > >
> > > Well, by end user here, I meant the same person that had to edit
> > > topology.xml.
> > > Topology.xml is much simpler than this file and most of what is in
> > > shiro.ini here should never need to be seen - even by the
> administrators.
> > > Giving the ability to mess with the order of the chain and the like is
> > only
> > > asking for trouble and user problems.
> > > I think that we should limit the amount of time we have to support
> Shiro
> > > config file problems by keeping folks out as much as we can.
> > >
> > >
> > > >
> > > > To integrate additional back end hadoop service, following pieces of
> > > > information have to come from pointers from knox.ini to external
> > > > configuration files.
> > > >
> > > > 1. end point of the new hadoop servie back end
> > > > 2. rewriting rules for the new service
> > >
> > >
> > > > These can be specified in one file or split between 2 files with one
> of
> > > > them being topology file or its equivalent.
> > > > I have not shown all those details in the current shiro.ini.
> > > > But, you would get the idea.
> > > >
> > >
> > > But if you don't show all of that then it isn't showing the whole
> > picture.
> > > What are you currently doing in your POC for the services?
> > >
> > >
> > > >
> > > > Dilli
> > > >
> > > >
> > > > On Fri, Oct 25, 2013 at 8:50 AM, larry mccay <lm...@apache.org>
> > wrote:
> > > >
> > > > > Hi Dilli -
> > > > >
> > > > > Good work here!
> > > > >
> > > > > Question:
> > > > > 1. Is this complicated ini hand edited by the enduser in order to
> > > deploy
> > > > a
> > > > > given topology or have you changed the shiro contributor to write
> > this
> > > > out
> > > > > from a much more understandable config within the topology?
> > > > >
> > > > > 2. Where are the services configured for a given topology?
> > > > > 3. Did you rename all the existing filters to start with knox?
> > > > >
> > > > > A couple observations:
> > > > >
> > > > > 1. removing the use of gateway.xml means that we *require* Shiro
> for
> > > our
> > > > > servlet filter chaining
> > > > > 2. While comparing this to the current format of gateway.xml I can
> > > agree
> > > > > that this is easier to understand - however, I don't actually find
> > ini
> > > > > format simpler to comprehend and administer than XML or JSON. In
> > fact,
> > > > > comparing this to topology.xml which is all you really need to do
> for
> > > > > admin, I think the topology file is very easy to understand and
> > > > administer.
> > > > > 3. moving the rewrite rules out to not require coding is good but
> > > should
> > > > > probably not require that they be in this format and file
> > > > > 4. Someone that wants to leverage their own or another third party
> > > > solution
> > > > > is stuck squeezing it into Shiro
> > > > > 5. This would violate a principle of mine of "Is-a" vs "Has-a" -
> Knox
> > > is
> > > > > not a Shiro provider it can however *have* a Shiro provider
> > > > > 6. End users should never have to see rewrite rules and maybe a lot
> > of
> > > > what
> > > > > is in that ini. Principal mapping, authz policy, etc will
> eventually
> > > come
> > > > > from Zookeeper which should remove their need to be in this file
> for
> > > > those
> > > > > things. Maybe we can even get the LDAP config separately from there
> > as
> > > > well
> > > > > - though I still think it is best provided in a topology file with
> a
> > > much
> > > > > smaller config requirements and get pushed into shiro.ini. Then the
> > > user
> > > > > may never need to see - or at least open - this file.
> > > > >
> > > > > I continue to believe that we are best served by enabling a
> > deployment
> > > > that
> > > > > looks like this and encouraging the use of it as the preferred
> > > solution.
> > > > > However, we should continue to support the use of arbitrary
> providers
> > > > > through the use of gateway.xml. In most cases, gateway.xml can be
> > > > leveraged
> > > > > to only include the Shiro provider. In addition, all knowledge of
> > Shiro
> > > > > specifics should remain inside of that provider. If it leaks out
> then
> > > we
> > > > > are violating the separation of Is-a vs Has-a again.
> > > > >
> > > > > This also allows us the flexibility to add something before or
> after
> > > that
> > > > > provider for things that we find are not easily or appropriately
> done
> > > > > inside the security provider.
> > > > >
> > > > > "I think we can rename shiro.ini as knox.ini to make it explicit
> this
> > > is
> > > > > more about knox configuration than shiro library configuration."
> > > > >
> > > > > While I can understand the thinking of renaming the shiro.ini file
> to
> > > > > knox.ini, also violates the Is-a vs Has-a principle for me. We are
> > not
> > > a
> > > > > shiro security provider extension. We are a REST API Gateway for
> > Hadoop
> > > > > that has security providers - one of which is Shiro.
> > > > >
> > > > > "We are using shiro config file mechanism as simple, lightweight
> > > > depenency
> > > > > injection."
> > > > >
> > > > > Where is there dependency injection here? If you are referring to
> the
> > > > > configuration of the filter chain that is not really injection
> though
> > > it
> > > > is
> > > > > an admittedly simpler mechanism than the current state of the
> > > > contributors
> > > > > and our deployment machinery - which could be refactored and
> > improved.
> > > > >
> > > > > "This does not really tie us to using only Shiro authentication
> > > mechanism
> > > > > or
> > > > > authorization mechanisms.
> > > > >
> > > > > We have the choice of writing all our authentication or
> authorization
> > > in
> > > > > our own servlet filters or leverage filters from Shiro library,
> > Realms
> > > > from
> > > > > Shiro library or write your own Shiro filters or Shiro realms."
> > > > >
> > > > > The only change that this proposal makes to our flexibility is the
> > > > > inability to take Shiro out of the picture.
> > > > > We could have leveraged Shiro to do all these things with the
> current
> > > > > design as well.
> > > > > BUT, we have control over our own interception channel in our
> chains
> > > > today.
> > > > > This proposal takes a specific *security provider's* proprietary
> > filter
> > > > > chain and uses it as the only interceptor channel for the entire
> > > server.
> > > > > Just because they also had a need for one - for their security
> work -
> > > > > doesn't mean that we shouldn't have our own. The tail should not
> wag
> > > the
> > > > > dog. Shiro is not an application or server framework it is a
> security
> > > > > provider.
> > > > >
> > > > > "In most of the cases, when we want to integrate with a new Hadoop
> > back
> > > > end
> > > > > service, all that we have to do is specify the path to a file
> having
> > > > > rewrite rules for the service. Rest of the things in
> > > knox.ini(=shiro.ini)
> > > > > would remain same."
> > > > >
> > > > > This isn't really very clear to me. Do we still have services in a
> > > > > topology.xml file?
> > > > > I don't see where you specify the url for the backend services -
> so I
> > > > > assume there is still a topology file.
> > > > > How do we know where to dispatch the requests to?
> > > > >
> > > > > I do agree that this is a very good step forward in defining a
> > > preferred
> > > > > security provider model for Knox and that we should continue down
> > this
> > > > > road. We just need to do so carefully. If, over time, we have found
> > no
> > > > > reason to use the existing solution that we can consider removing
> it
> > > > > entirely.
> > > > >
> > > > > Thanks for all this work and insight, Dilli!
> > > > >
> > > > > --larry
> > > > >
> > > > >
> > > > > On Fri, Oct 25, 2013 at 10:27 AM, Dilli Arumugam
> > > > > <da...@hortonworks.com>wrote:
> > > > >
> > > > > > Have been trying out a few ideas with Knox and managed to get
> Knox
> > > > > running
> > > > > > with
> > > > > >
> > > > > > 1. WEB-INF/gateway.xml is completely removed
> > > > > > 2. WEB-INF/web.xml declares only shiro filter and a defautl
> servlet
> > > > > > 2. all filters are defined and injected using WEB-INF/shiro.ini
> > > > > >
> > > > > > To me that looks much simpler to comprehend and administer.
> > > > > > Agreed, this could be subjective.
> > > > > > Hence, seeking comments from community.
> > > > > >
> > > > > > Pasting web.xml and shiro.ini inline.
> > > > > >
> > > > > > web.xml
> > > > > > --------------
> > > > > >
> > > > > > <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> > > > > > <web-app xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="
> > > > > > http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="
> > > > > > http://java.sun.com/xml/ns/javaee
> > > > > > http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd">
> > > > > >
> > > > > > <listener>
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> <listener-class>org.apache.shiro.web.env.EnvironmentLoaderListener</listener-class>
> > > > > > </listener>
> > > > > >
> > > > > > <listener>
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> <listener-class>org.apache.hadoop.gateway.services.GatewayServicesContextListener</listener-class>
> > > > > > </listener>
> > > > > >
> > > > > > <listener>
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> <listener-class>org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletContextListener</listener-class>
> > > > > > </listener>
> > > > > >
> > > > > > <context-param>
> > > > > > <param-name>rewriteDescriptorLocation</param-name>
> > > > > > <param-value>/WEB-INF/rewrite.xml</param-value>
> > > > > > </context-param>
> > > > > >
> > > > > > <filter>
> > > > > > <filter-name>ShiroFilter</filter-name>
> > > > > >
> > > > >
> <filter-class>org.apache.shiro.web.servlet.ShiroFilter</filter-class>
> > > > > > </filter>
> > > > > >
> > > > > > <filter-mapping>
> > > > > > <filter-name>ShiroFilter</filter-name>
> > > > > > <url-pattern>/*</url-pattern>
> > > > > > </filter-mapping>
> > > > > >
> > > > > > <session-config>
> > > > > > <session-timeout>30</session-timeout>
> > > > > > </session-config>
> > > > > >
> > > > > > <servlet>
> > > > > > <servlet-name>errorservlet</servlet-name>
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> <servlet-class>org.apache.hadoop.gateway.filter.KnoxErrorServlet</servlet-class>
> > > > > > </servlet>
> > > > > >
> > > > > > <servlet-mapping>
> > > > > > <servlet-name>errorservlet</servlet-name>
> > > > > > <url-pattern>/*</url-pattern>
> > > > > > </servlet-mapping>
> > > > > >
> > > > > > </web-app>
> > > > > >
> > > > > >
> > > > > > shiro.ini
> > > > > > -------------
> > > > > >
> > > > > > [main]
> > > > > >
> > > > > > # define ldapRealm
> > > > > > ldapRealm=org.apache.shiro.realm.ldap.JndiLdapRealm
> > > > > > ldapRealm.contextFactory.authenticationMechanism=simple
> > > > > > ldapRealm.contextFactory.url=ldap://localhost:33389
> > > > > >
> > ldapRealm.userDnTemplate=uid={0},ou=people,dc=hadoop,dc=apache,dc=org
> > > > > >
> > > > > >
> > > > > > # define filter: knoxResponseCookieFilter
> > > > > > knoxResponseCookieFilter =
> > > > > > org.apache.hadoop.gateway.filter.ResponseCookieFilter
> > > > > > knoxResponseCookieFilter.enabled = true
> > > > > > knoxResponseCookieFilter.filterHeaders = rememberMe,
> > > hadoop.auth.cookie
> > > > > >
> > > > > > # define filter: knoxPrincipalMapper
> > > > > > knoxPrincipalMapper =
> > > > > org.apache.hadoop.gateway.filter.KnoxPrincipalMapper
> > > > > > knoxPrincipalMapper.enabled = true
> > > > > > knoxPrincipalMapper.userToUserMap = bob:guest, jon:bob
> > > > > > knoxPrincipalMapper.userToGroupMap = *:users, bob:admin
> > > > > >
> > > > > > # define filter: knoxIPTracker
> > > > > > knoxIPTracker =org.apache.hadoop.gateway.filter.KnoxIPTracker
> > > > > > knoxIPTracker.enabled = true
> > > > > >
> > > > > > # define filter: knoxAclAuthzFilter
> > > > > > knoxAclAuthzFilter =
> > > > org.apache.hadoop.gateway.filter.KnoxAclAuthzFilter
> > > > > > knoxAclAuthzFilter.enabled = true
> > > > > > knoxAclAuthzFilter.globalGroupAclMode = OR
> > > > > > knoxAclAuthzFilter.serviceGroupAclModeMap = /webhdfs/:OR,
> > > > /templeton/:OR
> > > > > > knoxAclAuthzFilter.serviceAclMap = /webhdfs/:user1 user2; users
> > > admin;
> > > > > > 127.*.*.*, /templeton/:user11 user12; users admin
> > > > > >
> > > > > > # define filter: javaSubjectMapper
> > > > > > javaSubjectMapper =
> > > > > > org.apache.hadoop.gateway.filter.PostAuthenticationFilter
> > > > > >
> > > > > > # define filter: knoxIdentityAsserter
> > > > > > knoxIdentityAsserter =
> > > > > > org.apache.hadoop.gateway.filter.KnoxIdentityAsserter
> > > > > > knoxIdentityAsserter.enabled = true
> > > > > >
> > > > > > # define filter: knoxUrlRewriter
> > > > > > knoxUrlRewriter =
> > > > > >
> > org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter
> > > > > >
> > > > > > knoxUrlRewriter.requestUrlMap =
> > > > > > /webhdfs/v1/:WEBHDFS/webhdfs/inbound/namenode/root,
> > > > > > /webhdfs/v1/[^~]+:WEBHDFS/webhdfs/inbound/namenode/file,
> > > > > > /webhdfs/v1/~:WEBHDFS/webhdfs/inbound/namenode/home,
> > > > > > /webhdfs/v1/~/.*:WEBHDFS/webhdfs/inbound/namenode/home/file,
> > > > > > /webhdfs/data/v1/.+:WEBHDFS/webhdfs/inbound/datanode
> > > > > >
> > > > > > knoxUrlRewriter.requestBodyMap =
> /oozie/:OOZIE/oozie/configuration,
> > > > > > /oozie/v1/.*:OOZIE/oozie/configuration,
> > > > > > /oozie/v2/.*:OOZIE/oozie/configuration
> > > > > >
> > > > > > knoxUrlRewriter.responseHeadersMap =
> > > > > > /webhdfs/v1/[^~]*:WEBHDFS/webhdfs/outbound/namenode/headers,
> > > > > > /webhdfs/v1/~/:WEBHDFS/webhdfs/outbound/namenode/headers,
> > > > > > /hbase/:WEBHBASE/webhbase/headers/outbound,
> > > > > > /hbase/[~/]*:WEBHBASE/webhbase/headers/outbound,
> > > > > > /hbase/status/cluster:WEBHBASE/webhbase/status/outbound,
> > > > > > /hbase/[^/]*/regions:WEBHBASE/webhbase/regions/outbound
> > > > > >
> > > > > > # define filter: knoxHttpDispatcher
> > > > > > knoxHttpDispatcher =
> > > > > org.apache.hadoop.gateway.dispatch.HttpClientDispatch
> > > > > > knoxHttpDispatcher.replayBufferSizeMap = webhdfs:4, templeton:8,
> > > > oozie:4
> > > > > >
> > > > > > [urls]
> > > > > > # you could choose to have a different chain of filter for
> > different
> > > > url
> > > > > > patterns
> > > > > > # so far Knox did not need it
> > > > > > /** = knoxResponseCookieFilter, authcBasic, knoxPrincipalMapper,
> > > > > > knoxIPTracker, knoxAclAuthzFilter, javaSubjectMapper,
> > > > > knoxIdentityAsserter,
> > > > > > knoxUrlRewriter, knoxHttpDispatcher
> > > > > >
> > > > > > # end of shiro.ini
> > > > > >
> > > > > > I think we can rename shiro.ini as knox.ini to make it explicit
> > this
> > > is
> > > > > > more about knox configuration than shiro library configuration.
> > > > > >
> > > > > >
> > > > > > We are using shiro config file mechanism as simple, lightweight
> > > > depenency
> > > > > > injection.
> > > > > >
> > > > > > This does not really tie us to using only Shiro authentication
> > > > mechanism
> > > > > or
> > > > > > authorization mechanisms.
> > > > > >
> > > > > > We have the choice of writing all our authentication or
> > authorization
> > > > in
> > > > > > our own servlet filters or leverage filters from Shiro library,
> > > Realms
> > > > > from
> > > > > > Shiro library or write your own Shiro filters or Shiro realms.
> > > > > >
> > > > > > In most of the cases, when we want to integrate with a new Hadoop
> > > back
> > > > > end
> > > > > > service, all that we have to do is specify the path to a file
> > having
> > > > > > rewrite rules for the service. Rest of the things in
> > > > knox.ini(=shiro.ini)
> > > > > > would remain same.
> > > > > >
> > > > > > Please review and comment.
> > > > > >
> > > > > >
> > > > > > Thanks
> > > > > > Dilli
> > > > > >
> > > > > > --
> > > > > > CONFIDENTIALITY NOTICE
> > > > > > NOTICE: This message is intended for the use of the individual or
> > > > entity
> > > > > to
> > > > > > which it is addressed and may contain information that is
> > > confidential,
> > > > > > privileged and exempt from disclosure under applicable law. If
> the
> > > > reader
> > > > > > of this message is not the intended recipient, you are hereby
> > > notified
> > > > > that
> > > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > > forwarding of this communication is strictly prohibited. If you
> > have
> > > > > > received this communication in error, please contact the sender
> > > > > immediately
> > > > > > and delete it from your system. Thank You.
> > > > > >
> > > > >
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > to
> > > > which it is addressed and may contain information that is
> confidential,
> > > > privileged and exempt from disclosure under applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > > >
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: some design thoughts on knox - simplify integrating new service,
configuration, dependency injection
Posted by larry mccay <la...@gmail.com>.
hmmmm, so, where does the dispatcher get the endpoint info from - the
topology file?
That isn't even in WEB-INF.
On Fri, Oct 25, 2013 at 12:30 PM, Dilli Arumugam
<da...@hortonworks.com>wrote:
> The POC is done starting with a knox web app already deployed.
> Then,
>
> 1. remove WEB-INF/gateway.xml
> 2. update WEB-INF/web.xml
> 3. update WEB-INF/shiro.ini
>
> So, the web app was first deployed using current knox contributor
> framework.
> Have not spent time to simulate the equivalent of contribution framework.
>
> Thanks
> Dilli
>
>
> On Fri, Oct 25, 2013 at 9:23 AM, larry mccay <la...@gmail.com>
> wrote:
>
> > On Fri, Oct 25, 2013 at 12:03 PM, Dilli Arumugam
> > <da...@hortonworks.com>wrote:
> >
> > > Thanks Larry for the review and comments.
> > >
> > > The proposed knox.ini is meant to be edited by Knox system
> administrator
> > > until we get to have a admin console. This is not meant to be edited by
> > end
> > > user.
> > >
> >
> > Well, by end user here, I meant the same person that had to edit
> > topology.xml.
> > Topology.xml is much simpler than this file and most of what is in
> > shiro.ini here should never need to be seen - even by the administrators.
> > Giving the ability to mess with the order of the chain and the like is
> only
> > asking for trouble and user problems.
> > I think that we should limit the amount of time we have to support Shiro
> > config file problems by keeping folks out as much as we can.
> >
> >
> > >
> > > To integrate additional back end hadoop service, following pieces of
> > > information have to come from pointers from knox.ini to external
> > > configuration files.
> > >
> > > 1. end point of the new hadoop servie back end
> > > 2. rewriting rules for the new service
> >
> >
> > > These can be specified in one file or split between 2 files with one of
> > > them being topology file or its equivalent.
> > > I have not shown all those details in the current shiro.ini.
> > > But, you would get the idea.
> > >
> >
> > But if you don't show all of that then it isn't showing the whole
> picture.
> > What are you currently doing in your POC for the services?
> >
> >
> > >
> > > Dilli
> > >
> > >
> > > On Fri, Oct 25, 2013 at 8:50 AM, larry mccay <lm...@apache.org>
> wrote:
> > >
> > > > Hi Dilli -
> > > >
> > > > Good work here!
> > > >
> > > > Question:
> > > > 1. Is this complicated ini hand edited by the enduser in order to
> > deploy
> > > a
> > > > given topology or have you changed the shiro contributor to write
> this
> > > out
> > > > from a much more understandable config within the topology?
> > > >
> > > > 2. Where are the services configured for a given topology?
> > > > 3. Did you rename all the existing filters to start with knox?
> > > >
> > > > A couple observations:
> > > >
> > > > 1. removing the use of gateway.xml means that we *require* Shiro for
> > our
> > > > servlet filter chaining
> > > > 2. While comparing this to the current format of gateway.xml I can
> > agree
> > > > that this is easier to understand - however, I don't actually find
> ini
> > > > format simpler to comprehend and administer than XML or JSON. In
> fact,
> > > > comparing this to topology.xml which is all you really need to do for
> > > > admin, I think the topology file is very easy to understand and
> > > administer.
> > > > 3. moving the rewrite rules out to not require coding is good but
> > should
> > > > probably not require that they be in this format and file
> > > > 4. Someone that wants to leverage their own or another third party
> > > solution
> > > > is stuck squeezing it into Shiro
> > > > 5. This would violate a principle of mine of "Is-a" vs "Has-a" - Knox
> > is
> > > > not a Shiro provider it can however *have* a Shiro provider
> > > > 6. End users should never have to see rewrite rules and maybe a lot
> of
> > > what
> > > > is in that ini. Principal mapping, authz policy, etc will eventually
> > come
> > > > from Zookeeper which should remove their need to be in this file for
> > > those
> > > > things. Maybe we can even get the LDAP config separately from there
> as
> > > well
> > > > - though I still think it is best provided in a topology file with a
> > much
> > > > smaller config requirements and get pushed into shiro.ini. Then the
> > user
> > > > may never need to see - or at least open - this file.
> > > >
> > > > I continue to believe that we are best served by enabling a
> deployment
> > > that
> > > > looks like this and encouraging the use of it as the preferred
> > solution.
> > > > However, we should continue to support the use of arbitrary providers
> > > > through the use of gateway.xml. In most cases, gateway.xml can be
> > > leveraged
> > > > to only include the Shiro provider. In addition, all knowledge of
> Shiro
> > > > specifics should remain inside of that provider. If it leaks out then
> > we
> > > > are violating the separation of Is-a vs Has-a again.
> > > >
> > > > This also allows us the flexibility to add something before or after
> > that
> > > > provider for things that we find are not easily or appropriately done
> > > > inside the security provider.
> > > >
> > > > "I think we can rename shiro.ini as knox.ini to make it explicit this
> > is
> > > > more about knox configuration than shiro library configuration."
> > > >
> > > > While I can understand the thinking of renaming the shiro.ini file to
> > > > knox.ini, also violates the Is-a vs Has-a principle for me. We are
> not
> > a
> > > > shiro security provider extension. We are a REST API Gateway for
> Hadoop
> > > > that has security providers - one of which is Shiro.
> > > >
> > > > "We are using shiro config file mechanism as simple, lightweight
> > > depenency
> > > > injection."
> > > >
> > > > Where is there dependency injection here? If you are referring to the
> > > > configuration of the filter chain that is not really injection though
> > it
> > > is
> > > > an admittedly simpler mechanism than the current state of the
> > > contributors
> > > > and our deployment machinery - which could be refactored and
> improved.
> > > >
> > > > "This does not really tie us to using only Shiro authentication
> > mechanism
> > > > or
> > > > authorization mechanisms.
> > > >
> > > > We have the choice of writing all our authentication or authorization
> > in
> > > > our own servlet filters or leverage filters from Shiro library,
> Realms
> > > from
> > > > Shiro library or write your own Shiro filters or Shiro realms."
> > > >
> > > > The only change that this proposal makes to our flexibility is the
> > > > inability to take Shiro out of the picture.
> > > > We could have leveraged Shiro to do all these things with the current
> > > > design as well.
> > > > BUT, we have control over our own interception channel in our chains
> > > today.
> > > > This proposal takes a specific *security provider's* proprietary
> filter
> > > > chain and uses it as the only interceptor channel for the entire
> > server.
> > > > Just because they also had a need for one - for their security work -
> > > > doesn't mean that we shouldn't have our own. The tail should not wag
> > the
> > > > dog. Shiro is not an application or server framework it is a security
> > > > provider.
> > > >
> > > > "In most of the cases, when we want to integrate with a new Hadoop
> back
> > > end
> > > > service, all that we have to do is specify the path to a file having
> > > > rewrite rules for the service. Rest of the things in
> > knox.ini(=shiro.ini)
> > > > would remain same."
> > > >
> > > > This isn't really very clear to me. Do we still have services in a
> > > > topology.xml file?
> > > > I don't see where you specify the url for the backend services - so I
> > > > assume there is still a topology file.
> > > > How do we know where to dispatch the requests to?
> > > >
> > > > I do agree that this is a very good step forward in defining a
> > preferred
> > > > security provider model for Knox and that we should continue down
> this
> > > > road. We just need to do so carefully. If, over time, we have found
> no
> > > > reason to use the existing solution that we can consider removing it
> > > > entirely.
> > > >
> > > > Thanks for all this work and insight, Dilli!
> > > >
> > > > --larry
> > > >
> > > >
> > > > On Fri, Oct 25, 2013 at 10:27 AM, Dilli Arumugam
> > > > <da...@hortonworks.com>wrote:
> > > >
> > > > > Have been trying out a few ideas with Knox and managed to get Knox
> > > > running
> > > > > with
> > > > >
> > > > > 1. WEB-INF/gateway.xml is completely removed
> > > > > 2. WEB-INF/web.xml declares only shiro filter and a defautl servlet
> > > > > 2. all filters are defined and injected using WEB-INF/shiro.ini
> > > > >
> > > > > To me that looks much simpler to comprehend and administer.
> > > > > Agreed, this could be subjective.
> > > > > Hence, seeking comments from community.
> > > > >
> > > > > Pasting web.xml and shiro.ini inline.
> > > > >
> > > > > web.xml
> > > > > --------------
> > > > >
> > > > > <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> > > > > <web-app xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="
> > > > > http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="
> > > > > http://java.sun.com/xml/ns/javaee
> > > > > http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd">
> > > > >
> > > > > <listener>
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> <listener-class>org.apache.shiro.web.env.EnvironmentLoaderListener</listener-class>
> > > > > </listener>
> > > > >
> > > > > <listener>
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> <listener-class>org.apache.hadoop.gateway.services.GatewayServicesContextListener</listener-class>
> > > > > </listener>
> > > > >
> > > > > <listener>
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> <listener-class>org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletContextListener</listener-class>
> > > > > </listener>
> > > > >
> > > > > <context-param>
> > > > > <param-name>rewriteDescriptorLocation</param-name>
> > > > > <param-value>/WEB-INF/rewrite.xml</param-value>
> > > > > </context-param>
> > > > >
> > > > > <filter>
> > > > > <filter-name>ShiroFilter</filter-name>
> > > > >
> > > > <filter-class>org.apache.shiro.web.servlet.ShiroFilter</filter-class>
> > > > > </filter>
> > > > >
> > > > > <filter-mapping>
> > > > > <filter-name>ShiroFilter</filter-name>
> > > > > <url-pattern>/*</url-pattern>
> > > > > </filter-mapping>
> > > > >
> > > > > <session-config>
> > > > > <session-timeout>30</session-timeout>
> > > > > </session-config>
> > > > >
> > > > > <servlet>
> > > > > <servlet-name>errorservlet</servlet-name>
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> <servlet-class>org.apache.hadoop.gateway.filter.KnoxErrorServlet</servlet-class>
> > > > > </servlet>
> > > > >
> > > > > <servlet-mapping>
> > > > > <servlet-name>errorservlet</servlet-name>
> > > > > <url-pattern>/*</url-pattern>
> > > > > </servlet-mapping>
> > > > >
> > > > > </web-app>
> > > > >
> > > > >
> > > > > shiro.ini
> > > > > -------------
> > > > >
> > > > > [main]
> > > > >
> > > > > # define ldapRealm
> > > > > ldapRealm=org.apache.shiro.realm.ldap.JndiLdapRealm
> > > > > ldapRealm.contextFactory.authenticationMechanism=simple
> > > > > ldapRealm.contextFactory.url=ldap://localhost:33389
> > > > >
> ldapRealm.userDnTemplate=uid={0},ou=people,dc=hadoop,dc=apache,dc=org
> > > > >
> > > > >
> > > > > # define filter: knoxResponseCookieFilter
> > > > > knoxResponseCookieFilter =
> > > > > org.apache.hadoop.gateway.filter.ResponseCookieFilter
> > > > > knoxResponseCookieFilter.enabled = true
> > > > > knoxResponseCookieFilter.filterHeaders = rememberMe,
> > hadoop.auth.cookie
> > > > >
> > > > > # define filter: knoxPrincipalMapper
> > > > > knoxPrincipalMapper =
> > > > org.apache.hadoop.gateway.filter.KnoxPrincipalMapper
> > > > > knoxPrincipalMapper.enabled = true
> > > > > knoxPrincipalMapper.userToUserMap = bob:guest, jon:bob
> > > > > knoxPrincipalMapper.userToGroupMap = *:users, bob:admin
> > > > >
> > > > > # define filter: knoxIPTracker
> > > > > knoxIPTracker =org.apache.hadoop.gateway.filter.KnoxIPTracker
> > > > > knoxIPTracker.enabled = true
> > > > >
> > > > > # define filter: knoxAclAuthzFilter
> > > > > knoxAclAuthzFilter =
> > > org.apache.hadoop.gateway.filter.KnoxAclAuthzFilter
> > > > > knoxAclAuthzFilter.enabled = true
> > > > > knoxAclAuthzFilter.globalGroupAclMode = OR
> > > > > knoxAclAuthzFilter.serviceGroupAclModeMap = /webhdfs/:OR,
> > > /templeton/:OR
> > > > > knoxAclAuthzFilter.serviceAclMap = /webhdfs/:user1 user2; users
> > admin;
> > > > > 127.*.*.*, /templeton/:user11 user12; users admin
> > > > >
> > > > > # define filter: javaSubjectMapper
> > > > > javaSubjectMapper =
> > > > > org.apache.hadoop.gateway.filter.PostAuthenticationFilter
> > > > >
> > > > > # define filter: knoxIdentityAsserter
> > > > > knoxIdentityAsserter =
> > > > > org.apache.hadoop.gateway.filter.KnoxIdentityAsserter
> > > > > knoxIdentityAsserter.enabled = true
> > > > >
> > > > > # define filter: knoxUrlRewriter
> > > > > knoxUrlRewriter =
> > > > >
> org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter
> > > > >
> > > > > knoxUrlRewriter.requestUrlMap =
> > > > > /webhdfs/v1/:WEBHDFS/webhdfs/inbound/namenode/root,
> > > > > /webhdfs/v1/[^~]+:WEBHDFS/webhdfs/inbound/namenode/file,
> > > > > /webhdfs/v1/~:WEBHDFS/webhdfs/inbound/namenode/home,
> > > > > /webhdfs/v1/~/.*:WEBHDFS/webhdfs/inbound/namenode/home/file,
> > > > > /webhdfs/data/v1/.+:WEBHDFS/webhdfs/inbound/datanode
> > > > >
> > > > > knoxUrlRewriter.requestBodyMap = /oozie/:OOZIE/oozie/configuration,
> > > > > /oozie/v1/.*:OOZIE/oozie/configuration,
> > > > > /oozie/v2/.*:OOZIE/oozie/configuration
> > > > >
> > > > > knoxUrlRewriter.responseHeadersMap =
> > > > > /webhdfs/v1/[^~]*:WEBHDFS/webhdfs/outbound/namenode/headers,
> > > > > /webhdfs/v1/~/:WEBHDFS/webhdfs/outbound/namenode/headers,
> > > > > /hbase/:WEBHBASE/webhbase/headers/outbound,
> > > > > /hbase/[~/]*:WEBHBASE/webhbase/headers/outbound,
> > > > > /hbase/status/cluster:WEBHBASE/webhbase/status/outbound,
> > > > > /hbase/[^/]*/regions:WEBHBASE/webhbase/regions/outbound
> > > > >
> > > > > # define filter: knoxHttpDispatcher
> > > > > knoxHttpDispatcher =
> > > > org.apache.hadoop.gateway.dispatch.HttpClientDispatch
> > > > > knoxHttpDispatcher.replayBufferSizeMap = webhdfs:4, templeton:8,
> > > oozie:4
> > > > >
> > > > > [urls]
> > > > > # you could choose to have a different chain of filter for
> different
> > > url
> > > > > patterns
> > > > > # so far Knox did not need it
> > > > > /** = knoxResponseCookieFilter, authcBasic, knoxPrincipalMapper,
> > > > > knoxIPTracker, knoxAclAuthzFilter, javaSubjectMapper,
> > > > knoxIdentityAsserter,
> > > > > knoxUrlRewriter, knoxHttpDispatcher
> > > > >
> > > > > # end of shiro.ini
> > > > >
> > > > > I think we can rename shiro.ini as knox.ini to make it explicit
> this
> > is
> > > > > more about knox configuration than shiro library configuration.
> > > > >
> > > > >
> > > > > We are using shiro config file mechanism as simple, lightweight
> > > depenency
> > > > > injection.
> > > > >
> > > > > This does not really tie us to using only Shiro authentication
> > > mechanism
> > > > or
> > > > > authorization mechanisms.
> > > > >
> > > > > We have the choice of writing all our authentication or
> authorization
> > > in
> > > > > our own servlet filters or leverage filters from Shiro library,
> > Realms
> > > > from
> > > > > Shiro library or write your own Shiro filters or Shiro realms.
> > > > >
> > > > > In most of the cases, when we want to integrate with a new Hadoop
> > back
> > > > end
> > > > > service, all that we have to do is specify the path to a file
> having
> > > > > rewrite rules for the service. Rest of the things in
> > > knox.ini(=shiro.ini)
> > > > > would remain same.
> > > > >
> > > > > Please review and comment.
> > > > >
> > > > >
> > > > > Thanks
> > > > > Dilli
> > > > >
> > > > > --
> > > > > CONFIDENTIALITY NOTICE
> > > > > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > > to
> > > > > which it is addressed and may contain information that is
> > confidential,
> > > > > privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > > > of this message is not the intended recipient, you are hereby
> > notified
> > > > that
> > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > forwarding of this communication is strictly prohibited. If you
> have
> > > > > received this communication in error, please contact the sender
> > > > immediately
> > > > > and delete it from your system. Thank You.
> > > > >
> > > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: some design thoughts on knox - simplify integrating new service,
configuration, dependency injection
Posted by Dilli Arumugam <da...@hortonworks.com>.
The POC is done starting with a knox web app already deployed.
Then,
1. remove WEB-INF/gateway.xml
2. update WEB-INF/web.xml
3. update WEB-INF/shiro.ini
So, the web app was first deployed using current knox contributor
framework.
Have not spent time to simulate the equivalent of contribution framework.
Thanks
Dilli
On Fri, Oct 25, 2013 at 9:23 AM, larry mccay <la...@gmail.com> wrote:
> On Fri, Oct 25, 2013 at 12:03 PM, Dilli Arumugam
> <da...@hortonworks.com>wrote:
>
> > Thanks Larry for the review and comments.
> >
> > The proposed knox.ini is meant to be edited by Knox system administrator
> > until we get to have a admin console. This is not meant to be edited by
> end
> > user.
> >
>
> Well, by end user here, I meant the same person that had to edit
> topology.xml.
> Topology.xml is much simpler than this file and most of what is in
> shiro.ini here should never need to be seen - even by the administrators.
> Giving the ability to mess with the order of the chain and the like is only
> asking for trouble and user problems.
> I think that we should limit the amount of time we have to support Shiro
> config file problems by keeping folks out as much as we can.
>
>
> >
> > To integrate additional back end hadoop service, following pieces of
> > information have to come from pointers from knox.ini to external
> > configuration files.
> >
> > 1. end point of the new hadoop servie back end
> > 2. rewriting rules for the new service
>
>
> > These can be specified in one file or split between 2 files with one of
> > them being topology file or its equivalent.
> > I have not shown all those details in the current shiro.ini.
> > But, you would get the idea.
> >
>
> But if you don't show all of that then it isn't showing the whole picture.
> What are you currently doing in your POC for the services?
>
>
> >
> > Dilli
> >
> >
> > On Fri, Oct 25, 2013 at 8:50 AM, larry mccay <lm...@apache.org> wrote:
> >
> > > Hi Dilli -
> > >
> > > Good work here!
> > >
> > > Question:
> > > 1. Is this complicated ini hand edited by the enduser in order to
> deploy
> > a
> > > given topology or have you changed the shiro contributor to write this
> > out
> > > from a much more understandable config within the topology?
> > >
> > > 2. Where are the services configured for a given topology?
> > > 3. Did you rename all the existing filters to start with knox?
> > >
> > > A couple observations:
> > >
> > > 1. removing the use of gateway.xml means that we *require* Shiro for
> our
> > > servlet filter chaining
> > > 2. While comparing this to the current format of gateway.xml I can
> agree
> > > that this is easier to understand - however, I don't actually find ini
> > > format simpler to comprehend and administer than XML or JSON. In fact,
> > > comparing this to topology.xml which is all you really need to do for
> > > admin, I think the topology file is very easy to understand and
> > administer.
> > > 3. moving the rewrite rules out to not require coding is good but
> should
> > > probably not require that they be in this format and file
> > > 4. Someone that wants to leverage their own or another third party
> > solution
> > > is stuck squeezing it into Shiro
> > > 5. This would violate a principle of mine of "Is-a" vs "Has-a" - Knox
> is
> > > not a Shiro provider it can however *have* a Shiro provider
> > > 6. End users should never have to see rewrite rules and maybe a lot of
> > what
> > > is in that ini. Principal mapping, authz policy, etc will eventually
> come
> > > from Zookeeper which should remove their need to be in this file for
> > those
> > > things. Maybe we can even get the LDAP config separately from there as
> > well
> > > - though I still think it is best provided in a topology file with a
> much
> > > smaller config requirements and get pushed into shiro.ini. Then the
> user
> > > may never need to see - or at least open - this file.
> > >
> > > I continue to believe that we are best served by enabling a deployment
> > that
> > > looks like this and encouraging the use of it as the preferred
> solution.
> > > However, we should continue to support the use of arbitrary providers
> > > through the use of gateway.xml. In most cases, gateway.xml can be
> > leveraged
> > > to only include the Shiro provider. In addition, all knowledge of Shiro
> > > specifics should remain inside of that provider. If it leaks out then
> we
> > > are violating the separation of Is-a vs Has-a again.
> > >
> > > This also allows us the flexibility to add something before or after
> that
> > > provider for things that we find are not easily or appropriately done
> > > inside the security provider.
> > >
> > > "I think we can rename shiro.ini as knox.ini to make it explicit this
> is
> > > more about knox configuration than shiro library configuration."
> > >
> > > While I can understand the thinking of renaming the shiro.ini file to
> > > knox.ini, also violates the Is-a vs Has-a principle for me. We are not
> a
> > > shiro security provider extension. We are a REST API Gateway for Hadoop
> > > that has security providers - one of which is Shiro.
> > >
> > > "We are using shiro config file mechanism as simple, lightweight
> > depenency
> > > injection."
> > >
> > > Where is there dependency injection here? If you are referring to the
> > > configuration of the filter chain that is not really injection though
> it
> > is
> > > an admittedly simpler mechanism than the current state of the
> > contributors
> > > and our deployment machinery - which could be refactored and improved.
> > >
> > > "This does not really tie us to using only Shiro authentication
> mechanism
> > > or
> > > authorization mechanisms.
> > >
> > > We have the choice of writing all our authentication or authorization
> in
> > > our own servlet filters or leverage filters from Shiro library, Realms
> > from
> > > Shiro library or write your own Shiro filters or Shiro realms."
> > >
> > > The only change that this proposal makes to our flexibility is the
> > > inability to take Shiro out of the picture.
> > > We could have leveraged Shiro to do all these things with the current
> > > design as well.
> > > BUT, we have control over our own interception channel in our chains
> > today.
> > > This proposal takes a specific *security provider's* proprietary filter
> > > chain and uses it as the only interceptor channel for the entire
> server.
> > > Just because they also had a need for one - for their security work -
> > > doesn't mean that we shouldn't have our own. The tail should not wag
> the
> > > dog. Shiro is not an application or server framework it is a security
> > > provider.
> > >
> > > "In most of the cases, when we want to integrate with a new Hadoop back
> > end
> > > service, all that we have to do is specify the path to a file having
> > > rewrite rules for the service. Rest of the things in
> knox.ini(=shiro.ini)
> > > would remain same."
> > >
> > > This isn't really very clear to me. Do we still have services in a
> > > topology.xml file?
> > > I don't see where you specify the url for the backend services - so I
> > > assume there is still a topology file.
> > > How do we know where to dispatch the requests to?
> > >
> > > I do agree that this is a very good step forward in defining a
> preferred
> > > security provider model for Knox and that we should continue down this
> > > road. We just need to do so carefully. If, over time, we have found no
> > > reason to use the existing solution that we can consider removing it
> > > entirely.
> > >
> > > Thanks for all this work and insight, Dilli!
> > >
> > > --larry
> > >
> > >
> > > On Fri, Oct 25, 2013 at 10:27 AM, Dilli Arumugam
> > > <da...@hortonworks.com>wrote:
> > >
> > > > Have been trying out a few ideas with Knox and managed to get Knox
> > > running
> > > > with
> > > >
> > > > 1. WEB-INF/gateway.xml is completely removed
> > > > 2. WEB-INF/web.xml declares only shiro filter and a defautl servlet
> > > > 2. all filters are defined and injected using WEB-INF/shiro.ini
> > > >
> > > > To me that looks much simpler to comprehend and administer.
> > > > Agreed, this could be subjective.
> > > > Hence, seeking comments from community.
> > > >
> > > > Pasting web.xml and shiro.ini inline.
> > > >
> > > > web.xml
> > > > --------------
> > > >
> > > > <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> > > > <web-app xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="
> > > > http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="
> > > > http://java.sun.com/xml/ns/javaee
> > > > http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd">
> > > >
> > > > <listener>
> > > >
> > > >
> > > >
> > >
> >
> <listener-class>org.apache.shiro.web.env.EnvironmentLoaderListener</listener-class>
> > > > </listener>
> > > >
> > > > <listener>
> > > >
> > > >
> > > >
> > >
> >
> <listener-class>org.apache.hadoop.gateway.services.GatewayServicesContextListener</listener-class>
> > > > </listener>
> > > >
> > > > <listener>
> > > >
> > > >
> > > >
> > >
> >
> <listener-class>org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletContextListener</listener-class>
> > > > </listener>
> > > >
> > > > <context-param>
> > > > <param-name>rewriteDescriptorLocation</param-name>
> > > > <param-value>/WEB-INF/rewrite.xml</param-value>
> > > > </context-param>
> > > >
> > > > <filter>
> > > > <filter-name>ShiroFilter</filter-name>
> > > >
> > > <filter-class>org.apache.shiro.web.servlet.ShiroFilter</filter-class>
> > > > </filter>
> > > >
> > > > <filter-mapping>
> > > > <filter-name>ShiroFilter</filter-name>
> > > > <url-pattern>/*</url-pattern>
> > > > </filter-mapping>
> > > >
> > > > <session-config>
> > > > <session-timeout>30</session-timeout>
> > > > </session-config>
> > > >
> > > > <servlet>
> > > > <servlet-name>errorservlet</servlet-name>
> > > >
> > > >
> > > >
> > >
> >
> <servlet-class>org.apache.hadoop.gateway.filter.KnoxErrorServlet</servlet-class>
> > > > </servlet>
> > > >
> > > > <servlet-mapping>
> > > > <servlet-name>errorservlet</servlet-name>
> > > > <url-pattern>/*</url-pattern>
> > > > </servlet-mapping>
> > > >
> > > > </web-app>
> > > >
> > > >
> > > > shiro.ini
> > > > -------------
> > > >
> > > > [main]
> > > >
> > > > # define ldapRealm
> > > > ldapRealm=org.apache.shiro.realm.ldap.JndiLdapRealm
> > > > ldapRealm.contextFactory.authenticationMechanism=simple
> > > > ldapRealm.contextFactory.url=ldap://localhost:33389
> > > > ldapRealm.userDnTemplate=uid={0},ou=people,dc=hadoop,dc=apache,dc=org
> > > >
> > > >
> > > > # define filter: knoxResponseCookieFilter
> > > > knoxResponseCookieFilter =
> > > > org.apache.hadoop.gateway.filter.ResponseCookieFilter
> > > > knoxResponseCookieFilter.enabled = true
> > > > knoxResponseCookieFilter.filterHeaders = rememberMe,
> hadoop.auth.cookie
> > > >
> > > > # define filter: knoxPrincipalMapper
> > > > knoxPrincipalMapper =
> > > org.apache.hadoop.gateway.filter.KnoxPrincipalMapper
> > > > knoxPrincipalMapper.enabled = true
> > > > knoxPrincipalMapper.userToUserMap = bob:guest, jon:bob
> > > > knoxPrincipalMapper.userToGroupMap = *:users, bob:admin
> > > >
> > > > # define filter: knoxIPTracker
> > > > knoxIPTracker =org.apache.hadoop.gateway.filter.KnoxIPTracker
> > > > knoxIPTracker.enabled = true
> > > >
> > > > # define filter: knoxAclAuthzFilter
> > > > knoxAclAuthzFilter =
> > org.apache.hadoop.gateway.filter.KnoxAclAuthzFilter
> > > > knoxAclAuthzFilter.enabled = true
> > > > knoxAclAuthzFilter.globalGroupAclMode = OR
> > > > knoxAclAuthzFilter.serviceGroupAclModeMap = /webhdfs/:OR,
> > /templeton/:OR
> > > > knoxAclAuthzFilter.serviceAclMap = /webhdfs/:user1 user2; users
> admin;
> > > > 127.*.*.*, /templeton/:user11 user12; users admin
> > > >
> > > > # define filter: javaSubjectMapper
> > > > javaSubjectMapper =
> > > > org.apache.hadoop.gateway.filter.PostAuthenticationFilter
> > > >
> > > > # define filter: knoxIdentityAsserter
> > > > knoxIdentityAsserter =
> > > > org.apache.hadoop.gateway.filter.KnoxIdentityAsserter
> > > > knoxIdentityAsserter.enabled = true
> > > >
> > > > # define filter: knoxUrlRewriter
> > > > knoxUrlRewriter =
> > > > org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter
> > > >
> > > > knoxUrlRewriter.requestUrlMap =
> > > > /webhdfs/v1/:WEBHDFS/webhdfs/inbound/namenode/root,
> > > > /webhdfs/v1/[^~]+:WEBHDFS/webhdfs/inbound/namenode/file,
> > > > /webhdfs/v1/~:WEBHDFS/webhdfs/inbound/namenode/home,
> > > > /webhdfs/v1/~/.*:WEBHDFS/webhdfs/inbound/namenode/home/file,
> > > > /webhdfs/data/v1/.+:WEBHDFS/webhdfs/inbound/datanode
> > > >
> > > > knoxUrlRewriter.requestBodyMap = /oozie/:OOZIE/oozie/configuration,
> > > > /oozie/v1/.*:OOZIE/oozie/configuration,
> > > > /oozie/v2/.*:OOZIE/oozie/configuration
> > > >
> > > > knoxUrlRewriter.responseHeadersMap =
> > > > /webhdfs/v1/[^~]*:WEBHDFS/webhdfs/outbound/namenode/headers,
> > > > /webhdfs/v1/~/:WEBHDFS/webhdfs/outbound/namenode/headers,
> > > > /hbase/:WEBHBASE/webhbase/headers/outbound,
> > > > /hbase/[~/]*:WEBHBASE/webhbase/headers/outbound,
> > > > /hbase/status/cluster:WEBHBASE/webhbase/status/outbound,
> > > > /hbase/[^/]*/regions:WEBHBASE/webhbase/regions/outbound
> > > >
> > > > # define filter: knoxHttpDispatcher
> > > > knoxHttpDispatcher =
> > > org.apache.hadoop.gateway.dispatch.HttpClientDispatch
> > > > knoxHttpDispatcher.replayBufferSizeMap = webhdfs:4, templeton:8,
> > oozie:4
> > > >
> > > > [urls]
> > > > # you could choose to have a different chain of filter for different
> > url
> > > > patterns
> > > > # so far Knox did not need it
> > > > /** = knoxResponseCookieFilter, authcBasic, knoxPrincipalMapper,
> > > > knoxIPTracker, knoxAclAuthzFilter, javaSubjectMapper,
> > > knoxIdentityAsserter,
> > > > knoxUrlRewriter, knoxHttpDispatcher
> > > >
> > > > # end of shiro.ini
> > > >
> > > > I think we can rename shiro.ini as knox.ini to make it explicit this
> is
> > > > more about knox configuration than shiro library configuration.
> > > >
> > > >
> > > > We are using shiro config file mechanism as simple, lightweight
> > depenency
> > > > injection.
> > > >
> > > > This does not really tie us to using only Shiro authentication
> > mechanism
> > > or
> > > > authorization mechanisms.
> > > >
> > > > We have the choice of writing all our authentication or authorization
> > in
> > > > our own servlet filters or leverage filters from Shiro library,
> Realms
> > > from
> > > > Shiro library or write your own Shiro filters or Shiro realms.
> > > >
> > > > In most of the cases, when we want to integrate with a new Hadoop
> back
> > > end
> > > > service, all that we have to do is specify the path to a file having
> > > > rewrite rules for the service. Rest of the things in
> > knox.ini(=shiro.ini)
> > > > would remain same.
> > > >
> > > > Please review and comment.
> > > >
> > > >
> > > > Thanks
> > > > Dilli
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > to
> > > > which it is addressed and may contain information that is
> confidential,
> > > > privileged and exempt from disclosure under applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > > >
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: some design thoughts on knox - simplify integrating new service,
configuration, dependency injection
Posted by larry mccay <la...@gmail.com>.
On Fri, Oct 25, 2013 at 12:03 PM, Dilli Arumugam
<da...@hortonworks.com>wrote:
> Thanks Larry for the review and comments.
>
> The proposed knox.ini is meant to be edited by Knox system administrator
> until we get to have a admin console. This is not meant to be edited by end
> user.
>
Well, by end user here, I meant the same person that had to edit
topology.xml.
Topology.xml is much simpler than this file and most of what is in
shiro.ini here should never need to be seen - even by the administrators.
Giving the ability to mess with the order of the chain and the like is only
asking for trouble and user problems.
I think that we should limit the amount of time we have to support Shiro
config file problems by keeping folks out as much as we can.
>
> To integrate additional back end hadoop service, following pieces of
> information have to come from pointers from knox.ini to external
> configuration files.
>
> 1. end point of the new hadoop servie back end
> 2. rewriting rules for the new service
> These can be specified in one file or split between 2 files with one of
> them being topology file or its equivalent.
> I have not shown all those details in the current shiro.ini.
> But, you would get the idea.
>
But if you don't show all of that then it isn't showing the whole picture.
What are you currently doing in your POC for the services?
>
> Dilli
>
>
> On Fri, Oct 25, 2013 at 8:50 AM, larry mccay <lm...@apache.org> wrote:
>
> > Hi Dilli -
> >
> > Good work here!
> >
> > Question:
> > 1. Is this complicated ini hand edited by the enduser in order to deploy
> a
> > given topology or have you changed the shiro contributor to write this
> out
> > from a much more understandable config within the topology?
> >
> > 2. Where are the services configured for a given topology?
> > 3. Did you rename all the existing filters to start with knox?
> >
> > A couple observations:
> >
> > 1. removing the use of gateway.xml means that we *require* Shiro for our
> > servlet filter chaining
> > 2. While comparing this to the current format of gateway.xml I can agree
> > that this is easier to understand - however, I don't actually find ini
> > format simpler to comprehend and administer than XML or JSON. In fact,
> > comparing this to topology.xml which is all you really need to do for
> > admin, I think the topology file is very easy to understand and
> administer.
> > 3. moving the rewrite rules out to not require coding is good but should
> > probably not require that they be in this format and file
> > 4. Someone that wants to leverage their own or another third party
> solution
> > is stuck squeezing it into Shiro
> > 5. This would violate a principle of mine of "Is-a" vs "Has-a" - Knox is
> > not a Shiro provider it can however *have* a Shiro provider
> > 6. End users should never have to see rewrite rules and maybe a lot of
> what
> > is in that ini. Principal mapping, authz policy, etc will eventually come
> > from Zookeeper which should remove their need to be in this file for
> those
> > things. Maybe we can even get the LDAP config separately from there as
> well
> > - though I still think it is best provided in a topology file with a much
> > smaller config requirements and get pushed into shiro.ini. Then the user
> > may never need to see - or at least open - this file.
> >
> > I continue to believe that we are best served by enabling a deployment
> that
> > looks like this and encouraging the use of it as the preferred solution.
> > However, we should continue to support the use of arbitrary providers
> > through the use of gateway.xml. In most cases, gateway.xml can be
> leveraged
> > to only include the Shiro provider. In addition, all knowledge of Shiro
> > specifics should remain inside of that provider. If it leaks out then we
> > are violating the separation of Is-a vs Has-a again.
> >
> > This also allows us the flexibility to add something before or after that
> > provider for things that we find are not easily or appropriately done
> > inside the security provider.
> >
> > "I think we can rename shiro.ini as knox.ini to make it explicit this is
> > more about knox configuration than shiro library configuration."
> >
> > While I can understand the thinking of renaming the shiro.ini file to
> > knox.ini, also violates the Is-a vs Has-a principle for me. We are not a
> > shiro security provider extension. We are a REST API Gateway for Hadoop
> > that has security providers - one of which is Shiro.
> >
> > "We are using shiro config file mechanism as simple, lightweight
> depenency
> > injection."
> >
> > Where is there dependency injection here? If you are referring to the
> > configuration of the filter chain that is not really injection though it
> is
> > an admittedly simpler mechanism than the current state of the
> contributors
> > and our deployment machinery - which could be refactored and improved.
> >
> > "This does not really tie us to using only Shiro authentication mechanism
> > or
> > authorization mechanisms.
> >
> > We have the choice of writing all our authentication or authorization in
> > our own servlet filters or leverage filters from Shiro library, Realms
> from
> > Shiro library or write your own Shiro filters or Shiro realms."
> >
> > The only change that this proposal makes to our flexibility is the
> > inability to take Shiro out of the picture.
> > We could have leveraged Shiro to do all these things with the current
> > design as well.
> > BUT, we have control over our own interception channel in our chains
> today.
> > This proposal takes a specific *security provider's* proprietary filter
> > chain and uses it as the only interceptor channel for the entire server.
> > Just because they also had a need for one - for their security work -
> > doesn't mean that we shouldn't have our own. The tail should not wag the
> > dog. Shiro is not an application or server framework it is a security
> > provider.
> >
> > "In most of the cases, when we want to integrate with a new Hadoop back
> end
> > service, all that we have to do is specify the path to a file having
> > rewrite rules for the service. Rest of the things in knox.ini(=shiro.ini)
> > would remain same."
> >
> > This isn't really very clear to me. Do we still have services in a
> > topology.xml file?
> > I don't see where you specify the url for the backend services - so I
> > assume there is still a topology file.
> > How do we know where to dispatch the requests to?
> >
> > I do agree that this is a very good step forward in defining a preferred
> > security provider model for Knox and that we should continue down this
> > road. We just need to do so carefully. If, over time, we have found no
> > reason to use the existing solution that we can consider removing it
> > entirely.
> >
> > Thanks for all this work and insight, Dilli!
> >
> > --larry
> >
> >
> > On Fri, Oct 25, 2013 at 10:27 AM, Dilli Arumugam
> > <da...@hortonworks.com>wrote:
> >
> > > Have been trying out a few ideas with Knox and managed to get Knox
> > running
> > > with
> > >
> > > 1. WEB-INF/gateway.xml is completely removed
> > > 2. WEB-INF/web.xml declares only shiro filter and a defautl servlet
> > > 2. all filters are defined and injected using WEB-INF/shiro.ini
> > >
> > > To me that looks much simpler to comprehend and administer.
> > > Agreed, this could be subjective.
> > > Hence, seeking comments from community.
> > >
> > > Pasting web.xml and shiro.ini inline.
> > >
> > > web.xml
> > > --------------
> > >
> > > <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> > > <web-app xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="
> > > http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="
> > > http://java.sun.com/xml/ns/javaee
> > > http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd">
> > >
> > > <listener>
> > >
> > >
> > >
> >
> <listener-class>org.apache.shiro.web.env.EnvironmentLoaderListener</listener-class>
> > > </listener>
> > >
> > > <listener>
> > >
> > >
> > >
> >
> <listener-class>org.apache.hadoop.gateway.services.GatewayServicesContextListener</listener-class>
> > > </listener>
> > >
> > > <listener>
> > >
> > >
> > >
> >
> <listener-class>org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletContextListener</listener-class>
> > > </listener>
> > >
> > > <context-param>
> > > <param-name>rewriteDescriptorLocation</param-name>
> > > <param-value>/WEB-INF/rewrite.xml</param-value>
> > > </context-param>
> > >
> > > <filter>
> > > <filter-name>ShiroFilter</filter-name>
> > >
> > <filter-class>org.apache.shiro.web.servlet.ShiroFilter</filter-class>
> > > </filter>
> > >
> > > <filter-mapping>
> > > <filter-name>ShiroFilter</filter-name>
> > > <url-pattern>/*</url-pattern>
> > > </filter-mapping>
> > >
> > > <session-config>
> > > <session-timeout>30</session-timeout>
> > > </session-config>
> > >
> > > <servlet>
> > > <servlet-name>errorservlet</servlet-name>
> > >
> > >
> > >
> >
> <servlet-class>org.apache.hadoop.gateway.filter.KnoxErrorServlet</servlet-class>
> > > </servlet>
> > >
> > > <servlet-mapping>
> > > <servlet-name>errorservlet</servlet-name>
> > > <url-pattern>/*</url-pattern>
> > > </servlet-mapping>
> > >
> > > </web-app>
> > >
> > >
> > > shiro.ini
> > > -------------
> > >
> > > [main]
> > >
> > > # define ldapRealm
> > > ldapRealm=org.apache.shiro.realm.ldap.JndiLdapRealm
> > > ldapRealm.contextFactory.authenticationMechanism=simple
> > > ldapRealm.contextFactory.url=ldap://localhost:33389
> > > ldapRealm.userDnTemplate=uid={0},ou=people,dc=hadoop,dc=apache,dc=org
> > >
> > >
> > > # define filter: knoxResponseCookieFilter
> > > knoxResponseCookieFilter =
> > > org.apache.hadoop.gateway.filter.ResponseCookieFilter
> > > knoxResponseCookieFilter.enabled = true
> > > knoxResponseCookieFilter.filterHeaders = rememberMe, hadoop.auth.cookie
> > >
> > > # define filter: knoxPrincipalMapper
> > > knoxPrincipalMapper =
> > org.apache.hadoop.gateway.filter.KnoxPrincipalMapper
> > > knoxPrincipalMapper.enabled = true
> > > knoxPrincipalMapper.userToUserMap = bob:guest, jon:bob
> > > knoxPrincipalMapper.userToGroupMap = *:users, bob:admin
> > >
> > > # define filter: knoxIPTracker
> > > knoxIPTracker =org.apache.hadoop.gateway.filter.KnoxIPTracker
> > > knoxIPTracker.enabled = true
> > >
> > > # define filter: knoxAclAuthzFilter
> > > knoxAclAuthzFilter =
> org.apache.hadoop.gateway.filter.KnoxAclAuthzFilter
> > > knoxAclAuthzFilter.enabled = true
> > > knoxAclAuthzFilter.globalGroupAclMode = OR
> > > knoxAclAuthzFilter.serviceGroupAclModeMap = /webhdfs/:OR,
> /templeton/:OR
> > > knoxAclAuthzFilter.serviceAclMap = /webhdfs/:user1 user2; users admin;
> > > 127.*.*.*, /templeton/:user11 user12; users admin
> > >
> > > # define filter: javaSubjectMapper
> > > javaSubjectMapper =
> > > org.apache.hadoop.gateway.filter.PostAuthenticationFilter
> > >
> > > # define filter: knoxIdentityAsserter
> > > knoxIdentityAsserter =
> > > org.apache.hadoop.gateway.filter.KnoxIdentityAsserter
> > > knoxIdentityAsserter.enabled = true
> > >
> > > # define filter: knoxUrlRewriter
> > > knoxUrlRewriter =
> > > org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter
> > >
> > > knoxUrlRewriter.requestUrlMap =
> > > /webhdfs/v1/:WEBHDFS/webhdfs/inbound/namenode/root,
> > > /webhdfs/v1/[^~]+:WEBHDFS/webhdfs/inbound/namenode/file,
> > > /webhdfs/v1/~:WEBHDFS/webhdfs/inbound/namenode/home,
> > > /webhdfs/v1/~/.*:WEBHDFS/webhdfs/inbound/namenode/home/file,
> > > /webhdfs/data/v1/.+:WEBHDFS/webhdfs/inbound/datanode
> > >
> > > knoxUrlRewriter.requestBodyMap = /oozie/:OOZIE/oozie/configuration,
> > > /oozie/v1/.*:OOZIE/oozie/configuration,
> > > /oozie/v2/.*:OOZIE/oozie/configuration
> > >
> > > knoxUrlRewriter.responseHeadersMap =
> > > /webhdfs/v1/[^~]*:WEBHDFS/webhdfs/outbound/namenode/headers,
> > > /webhdfs/v1/~/:WEBHDFS/webhdfs/outbound/namenode/headers,
> > > /hbase/:WEBHBASE/webhbase/headers/outbound,
> > > /hbase/[~/]*:WEBHBASE/webhbase/headers/outbound,
> > > /hbase/status/cluster:WEBHBASE/webhbase/status/outbound,
> > > /hbase/[^/]*/regions:WEBHBASE/webhbase/regions/outbound
> > >
> > > # define filter: knoxHttpDispatcher
> > > knoxHttpDispatcher =
> > org.apache.hadoop.gateway.dispatch.HttpClientDispatch
> > > knoxHttpDispatcher.replayBufferSizeMap = webhdfs:4, templeton:8,
> oozie:4
> > >
> > > [urls]
> > > # you could choose to have a different chain of filter for different
> url
> > > patterns
> > > # so far Knox did not need it
> > > /** = knoxResponseCookieFilter, authcBasic, knoxPrincipalMapper,
> > > knoxIPTracker, knoxAclAuthzFilter, javaSubjectMapper,
> > knoxIdentityAsserter,
> > > knoxUrlRewriter, knoxHttpDispatcher
> > >
> > > # end of shiro.ini
> > >
> > > I think we can rename shiro.ini as knox.ini to make it explicit this is
> > > more about knox configuration than shiro library configuration.
> > >
> > >
> > > We are using shiro config file mechanism as simple, lightweight
> depenency
> > > injection.
> > >
> > > This does not really tie us to using only Shiro authentication
> mechanism
> > or
> > > authorization mechanisms.
> > >
> > > We have the choice of writing all our authentication or authorization
> in
> > > our own servlet filters or leverage filters from Shiro library, Realms
> > from
> > > Shiro library or write your own Shiro filters or Shiro realms.
> > >
> > > In most of the cases, when we want to integrate with a new Hadoop back
> > end
> > > service, all that we have to do is specify the path to a file having
> > > rewrite rules for the service. Rest of the things in
> knox.ini(=shiro.ini)
> > > would remain same.
> > >
> > > Please review and comment.
> > >
> > >
> > > Thanks
> > > Dilli
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: some design thoughts on knox - simplify integrating new service,
configuration, dependency injection
Posted by Dilli Arumugam <da...@hortonworks.com>.
Thanks Larry for the review and comments.
The proposed knox.ini is meant to be edited by Knox system administrator
until we get to have a admin console. This is not meant to be edited by end
user.
To integrate additional back end hadoop service, following pieces of
information have to come from pointers from knox.ini to external
configuration files.
1. end point of the new hadoop servie back end
2. rewriting rules for the new service
These can be specified in one file or split between 2 files with one of
them being topology file or its equivalent.
I have not shown all those details in the current shiro.ini.
But, you would get the idea.
Dilli
On Fri, Oct 25, 2013 at 8:50 AM, larry mccay <lm...@apache.org> wrote:
> Hi Dilli -
>
> Good work here!
>
> Question:
> 1. Is this complicated ini hand edited by the enduser in order to deploy a
> given topology or have you changed the shiro contributor to write this out
> from a much more understandable config within the topology?
>
> 2. Where are the services configured for a given topology?
> 3. Did you rename all the existing filters to start with knox?
>
> A couple observations:
>
> 1. removing the use of gateway.xml means that we *require* Shiro for our
> servlet filter chaining
> 2. While comparing this to the current format of gateway.xml I can agree
> that this is easier to understand - however, I don't actually find ini
> format simpler to comprehend and administer than XML or JSON. In fact,
> comparing this to topology.xml which is all you really need to do for
> admin, I think the topology file is very easy to understand and administer.
> 3. moving the rewrite rules out to not require coding is good but should
> probably not require that they be in this format and file
> 4. Someone that wants to leverage their own or another third party solution
> is stuck squeezing it into Shiro
> 5. This would violate a principle of mine of "Is-a" vs "Has-a" - Knox is
> not a Shiro provider it can however *have* a Shiro provider
> 6. End users should never have to see rewrite rules and maybe a lot of what
> is in that ini. Principal mapping, authz policy, etc will eventually come
> from Zookeeper which should remove their need to be in this file for those
> things. Maybe we can even get the LDAP config separately from there as well
> - though I still think it is best provided in a topology file with a much
> smaller config requirements and get pushed into shiro.ini. Then the user
> may never need to see - or at least open - this file.
>
> I continue to believe that we are best served by enabling a deployment that
> looks like this and encouraging the use of it as the preferred solution.
> However, we should continue to support the use of arbitrary providers
> through the use of gateway.xml. In most cases, gateway.xml can be leveraged
> to only include the Shiro provider. In addition, all knowledge of Shiro
> specifics should remain inside of that provider. If it leaks out then we
> are violating the separation of Is-a vs Has-a again.
>
> This also allows us the flexibility to add something before or after that
> provider for things that we find are not easily or appropriately done
> inside the security provider.
>
> "I think we can rename shiro.ini as knox.ini to make it explicit this is
> more about knox configuration than shiro library configuration."
>
> While I can understand the thinking of renaming the shiro.ini file to
> knox.ini, also violates the Is-a vs Has-a principle for me. We are not a
> shiro security provider extension. We are a REST API Gateway for Hadoop
> that has security providers - one of which is Shiro.
>
> "We are using shiro config file mechanism as simple, lightweight depenency
> injection."
>
> Where is there dependency injection here? If you are referring to the
> configuration of the filter chain that is not really injection though it is
> an admittedly simpler mechanism than the current state of the contributors
> and our deployment machinery - which could be refactored and improved.
>
> "This does not really tie us to using only Shiro authentication mechanism
> or
> authorization mechanisms.
>
> We have the choice of writing all our authentication or authorization in
> our own servlet filters or leverage filters from Shiro library, Realms from
> Shiro library or write your own Shiro filters or Shiro realms."
>
> The only change that this proposal makes to our flexibility is the
> inability to take Shiro out of the picture.
> We could have leveraged Shiro to do all these things with the current
> design as well.
> BUT, we have control over our own interception channel in our chains today.
> This proposal takes a specific *security provider's* proprietary filter
> chain and uses it as the only interceptor channel for the entire server.
> Just because they also had a need for one - for their security work -
> doesn't mean that we shouldn't have our own. The tail should not wag the
> dog. Shiro is not an application or server framework it is a security
> provider.
>
> "In most of the cases, when we want to integrate with a new Hadoop back end
> service, all that we have to do is specify the path to a file having
> rewrite rules for the service. Rest of the things in knox.ini(=shiro.ini)
> would remain same."
>
> This isn't really very clear to me. Do we still have services in a
> topology.xml file?
> I don't see where you specify the url for the backend services - so I
> assume there is still a topology file.
> How do we know where to dispatch the requests to?
>
> I do agree that this is a very good step forward in defining a preferred
> security provider model for Knox and that we should continue down this
> road. We just need to do so carefully. If, over time, we have found no
> reason to use the existing solution that we can consider removing it
> entirely.
>
> Thanks for all this work and insight, Dilli!
>
> --larry
>
>
> On Fri, Oct 25, 2013 at 10:27 AM, Dilli Arumugam
> <da...@hortonworks.com>wrote:
>
> > Have been trying out a few ideas with Knox and managed to get Knox
> running
> > with
> >
> > 1. WEB-INF/gateway.xml is completely removed
> > 2. WEB-INF/web.xml declares only shiro filter and a defautl servlet
> > 2. all filters are defined and injected using WEB-INF/shiro.ini
> >
> > To me that looks much simpler to comprehend and administer.
> > Agreed, this could be subjective.
> > Hence, seeking comments from community.
> >
> > Pasting web.xml and shiro.ini inline.
> >
> > web.xml
> > --------------
> >
> > <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> > <web-app xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="
> > http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="
> > http://java.sun.com/xml/ns/javaee
> > http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd">
> >
> > <listener>
> >
> >
> >
> <listener-class>org.apache.shiro.web.env.EnvironmentLoaderListener</listener-class>
> > </listener>
> >
> > <listener>
> >
> >
> >
> <listener-class>org.apache.hadoop.gateway.services.GatewayServicesContextListener</listener-class>
> > </listener>
> >
> > <listener>
> >
> >
> >
> <listener-class>org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletContextListener</listener-class>
> > </listener>
> >
> > <context-param>
> > <param-name>rewriteDescriptorLocation</param-name>
> > <param-value>/WEB-INF/rewrite.xml</param-value>
> > </context-param>
> >
> > <filter>
> > <filter-name>ShiroFilter</filter-name>
> >
> <filter-class>org.apache.shiro.web.servlet.ShiroFilter</filter-class>
> > </filter>
> >
> > <filter-mapping>
> > <filter-name>ShiroFilter</filter-name>
> > <url-pattern>/*</url-pattern>
> > </filter-mapping>
> >
> > <session-config>
> > <session-timeout>30</session-timeout>
> > </session-config>
> >
> > <servlet>
> > <servlet-name>errorservlet</servlet-name>
> >
> >
> >
> <servlet-class>org.apache.hadoop.gateway.filter.KnoxErrorServlet</servlet-class>
> > </servlet>
> >
> > <servlet-mapping>
> > <servlet-name>errorservlet</servlet-name>
> > <url-pattern>/*</url-pattern>
> > </servlet-mapping>
> >
> > </web-app>
> >
> >
> > shiro.ini
> > -------------
> >
> > [main]
> >
> > # define ldapRealm
> > ldapRealm=org.apache.shiro.realm.ldap.JndiLdapRealm
> > ldapRealm.contextFactory.authenticationMechanism=simple
> > ldapRealm.contextFactory.url=ldap://localhost:33389
> > ldapRealm.userDnTemplate=uid={0},ou=people,dc=hadoop,dc=apache,dc=org
> >
> >
> > # define filter: knoxResponseCookieFilter
> > knoxResponseCookieFilter =
> > org.apache.hadoop.gateway.filter.ResponseCookieFilter
> > knoxResponseCookieFilter.enabled = true
> > knoxResponseCookieFilter.filterHeaders = rememberMe, hadoop.auth.cookie
> >
> > # define filter: knoxPrincipalMapper
> > knoxPrincipalMapper =
> org.apache.hadoop.gateway.filter.KnoxPrincipalMapper
> > knoxPrincipalMapper.enabled = true
> > knoxPrincipalMapper.userToUserMap = bob:guest, jon:bob
> > knoxPrincipalMapper.userToGroupMap = *:users, bob:admin
> >
> > # define filter: knoxIPTracker
> > knoxIPTracker =org.apache.hadoop.gateway.filter.KnoxIPTracker
> > knoxIPTracker.enabled = true
> >
> > # define filter: knoxAclAuthzFilter
> > knoxAclAuthzFilter = org.apache.hadoop.gateway.filter.KnoxAclAuthzFilter
> > knoxAclAuthzFilter.enabled = true
> > knoxAclAuthzFilter.globalGroupAclMode = OR
> > knoxAclAuthzFilter.serviceGroupAclModeMap = /webhdfs/:OR, /templeton/:OR
> > knoxAclAuthzFilter.serviceAclMap = /webhdfs/:user1 user2; users admin;
> > 127.*.*.*, /templeton/:user11 user12; users admin
> >
> > # define filter: javaSubjectMapper
> > javaSubjectMapper =
> > org.apache.hadoop.gateway.filter.PostAuthenticationFilter
> >
> > # define filter: knoxIdentityAsserter
> > knoxIdentityAsserter =
> > org.apache.hadoop.gateway.filter.KnoxIdentityAsserter
> > knoxIdentityAsserter.enabled = true
> >
> > # define filter: knoxUrlRewriter
> > knoxUrlRewriter =
> > org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter
> >
> > knoxUrlRewriter.requestUrlMap =
> > /webhdfs/v1/:WEBHDFS/webhdfs/inbound/namenode/root,
> > /webhdfs/v1/[^~]+:WEBHDFS/webhdfs/inbound/namenode/file,
> > /webhdfs/v1/~:WEBHDFS/webhdfs/inbound/namenode/home,
> > /webhdfs/v1/~/.*:WEBHDFS/webhdfs/inbound/namenode/home/file,
> > /webhdfs/data/v1/.+:WEBHDFS/webhdfs/inbound/datanode
> >
> > knoxUrlRewriter.requestBodyMap = /oozie/:OOZIE/oozie/configuration,
> > /oozie/v1/.*:OOZIE/oozie/configuration,
> > /oozie/v2/.*:OOZIE/oozie/configuration
> >
> > knoxUrlRewriter.responseHeadersMap =
> > /webhdfs/v1/[^~]*:WEBHDFS/webhdfs/outbound/namenode/headers,
> > /webhdfs/v1/~/:WEBHDFS/webhdfs/outbound/namenode/headers,
> > /hbase/:WEBHBASE/webhbase/headers/outbound,
> > /hbase/[~/]*:WEBHBASE/webhbase/headers/outbound,
> > /hbase/status/cluster:WEBHBASE/webhbase/status/outbound,
> > /hbase/[^/]*/regions:WEBHBASE/webhbase/regions/outbound
> >
> > # define filter: knoxHttpDispatcher
> > knoxHttpDispatcher =
> org.apache.hadoop.gateway.dispatch.HttpClientDispatch
> > knoxHttpDispatcher.replayBufferSizeMap = webhdfs:4, templeton:8, oozie:4
> >
> > [urls]
> > # you could choose to have a different chain of filter for different url
> > patterns
> > # so far Knox did not need it
> > /** = knoxResponseCookieFilter, authcBasic, knoxPrincipalMapper,
> > knoxIPTracker, knoxAclAuthzFilter, javaSubjectMapper,
> knoxIdentityAsserter,
> > knoxUrlRewriter, knoxHttpDispatcher
> >
> > # end of shiro.ini
> >
> > I think we can rename shiro.ini as knox.ini to make it explicit this is
> > more about knox configuration than shiro library configuration.
> >
> >
> > We are using shiro config file mechanism as simple, lightweight depenency
> > injection.
> >
> > This does not really tie us to using only Shiro authentication mechanism
> or
> > authorization mechanisms.
> >
> > We have the choice of writing all our authentication or authorization in
> > our own servlet filters or leverage filters from Shiro library, Realms
> from
> > Shiro library or write your own Shiro filters or Shiro realms.
> >
> > In most of the cases, when we want to integrate with a new Hadoop back
> end
> > service, all that we have to do is specify the path to a file having
> > rewrite rules for the service. Rest of the things in knox.ini(=shiro.ini)
> > would remain same.
> >
> > Please review and comment.
> >
> >
> > Thanks
> > Dilli
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: some design thoughts on knox - simplify integrating new service,
configuration, dependency injection
Posted by larry mccay <lm...@apache.org>.
Hi Dilli -
Good work here!
Question:
1. Is this complicated ini hand edited by the enduser in order to deploy a
given topology or have you changed the shiro contributor to write this out
from a much more understandable config within the topology?
2. Where are the services configured for a given topology?
3. Did you rename all the existing filters to start with knox?
A couple observations:
1. removing the use of gateway.xml means that we *require* Shiro for our
servlet filter chaining
2. While comparing this to the current format of gateway.xml I can agree
that this is easier to understand - however, I don't actually find ini
format simpler to comprehend and administer than XML or JSON. In fact,
comparing this to topology.xml which is all you really need to do for
admin, I think the topology file is very easy to understand and administer.
3. moving the rewrite rules out to not require coding is good but should
probably not require that they be in this format and file
4. Someone that wants to leverage their own or another third party solution
is stuck squeezing it into Shiro
5. This would violate a principle of mine of "Is-a" vs "Has-a" - Knox is
not a Shiro provider it can however *have* a Shiro provider
6. End users should never have to see rewrite rules and maybe a lot of what
is in that ini. Principal mapping, authz policy, etc will eventually come
from Zookeeper which should remove their need to be in this file for those
things. Maybe we can even get the LDAP config separately from there as well
- though I still think it is best provided in a topology file with a much
smaller config requirements and get pushed into shiro.ini. Then the user
may never need to see - or at least open - this file.
I continue to believe that we are best served by enabling a deployment that
looks like this and encouraging the use of it as the preferred solution.
However, we should continue to support the use of arbitrary providers
through the use of gateway.xml. In most cases, gateway.xml can be leveraged
to only include the Shiro provider. In addition, all knowledge of Shiro
specifics should remain inside of that provider. If it leaks out then we
are violating the separation of Is-a vs Has-a again.
This also allows us the flexibility to add something before or after that
provider for things that we find are not easily or appropriately done
inside the security provider.
"I think we can rename shiro.ini as knox.ini to make it explicit this is
more about knox configuration than shiro library configuration."
While I can understand the thinking of renaming the shiro.ini file to
knox.ini, also violates the Is-a vs Has-a principle for me. We are not a
shiro security provider extension. We are a REST API Gateway for Hadoop
that has security providers - one of which is Shiro.
"We are using shiro config file mechanism as simple, lightweight depenency
injection."
Where is there dependency injection here? If you are referring to the
configuration of the filter chain that is not really injection though it is
an admittedly simpler mechanism than the current state of the contributors
and our deployment machinery - which could be refactored and improved.
"This does not really tie us to using only Shiro authentication mechanism or
authorization mechanisms.
We have the choice of writing all our authentication or authorization in
our own servlet filters or leverage filters from Shiro library, Realms from
Shiro library or write your own Shiro filters or Shiro realms."
The only change that this proposal makes to our flexibility is the
inability to take Shiro out of the picture.
We could have leveraged Shiro to do all these things with the current
design as well.
BUT, we have control over our own interception channel in our chains today.
This proposal takes a specific *security provider's* proprietary filter
chain and uses it as the only interceptor channel for the entire server.
Just because they also had a need for one - for their security work -
doesn't mean that we shouldn't have our own. The tail should not wag the
dog. Shiro is not an application or server framework it is a security
provider.
"In most of the cases, when we want to integrate with a new Hadoop back end
service, all that we have to do is specify the path to a file having
rewrite rules for the service. Rest of the things in knox.ini(=shiro.ini)
would remain same."
This isn't really very clear to me. Do we still have services in a
topology.xml file?
I don't see where you specify the url for the backend services - so I
assume there is still a topology file.
How do we know where to dispatch the requests to?
I do agree that this is a very good step forward in defining a preferred
security provider model for Knox and that we should continue down this
road. We just need to do so carefully. If, over time, we have found no
reason to use the existing solution that we can consider removing it
entirely.
Thanks for all this work and insight, Dilli!
--larry
On Fri, Oct 25, 2013 at 10:27 AM, Dilli Arumugam
<da...@hortonworks.com>wrote:
> Have been trying out a few ideas with Knox and managed to get Knox running
> with
>
> 1. WEB-INF/gateway.xml is completely removed
> 2. WEB-INF/web.xml declares only shiro filter and a defautl servlet
> 2. all filters are defined and injected using WEB-INF/shiro.ini
>
> To me that looks much simpler to comprehend and administer.
> Agreed, this could be subjective.
> Hence, seeking comments from community.
>
> Pasting web.xml and shiro.ini inline.
>
> web.xml
> --------------
>
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <web-app xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="
> http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="
> http://java.sun.com/xml/ns/javaee
> http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd">
>
> <listener>
>
>
> <listener-class>org.apache.shiro.web.env.EnvironmentLoaderListener</listener-class>
> </listener>
>
> <listener>
>
>
> <listener-class>org.apache.hadoop.gateway.services.GatewayServicesContextListener</listener-class>
> </listener>
>
> <listener>
>
>
> <listener-class>org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletContextListener</listener-class>
> </listener>
>
> <context-param>
> <param-name>rewriteDescriptorLocation</param-name>
> <param-value>/WEB-INF/rewrite.xml</param-value>
> </context-param>
>
> <filter>
> <filter-name>ShiroFilter</filter-name>
> <filter-class>org.apache.shiro.web.servlet.ShiroFilter</filter-class>
> </filter>
>
> <filter-mapping>
> <filter-name>ShiroFilter</filter-name>
> <url-pattern>/*</url-pattern>
> </filter-mapping>
>
> <session-config>
> <session-timeout>30</session-timeout>
> </session-config>
>
> <servlet>
> <servlet-name>errorservlet</servlet-name>
>
>
> <servlet-class>org.apache.hadoop.gateway.filter.KnoxErrorServlet</servlet-class>
> </servlet>
>
> <servlet-mapping>
> <servlet-name>errorservlet</servlet-name>
> <url-pattern>/*</url-pattern>
> </servlet-mapping>
>
> </web-app>
>
>
> shiro.ini
> -------------
>
> [main]
>
> # define ldapRealm
> ldapRealm=org.apache.shiro.realm.ldap.JndiLdapRealm
> ldapRealm.contextFactory.authenticationMechanism=simple
> ldapRealm.contextFactory.url=ldap://localhost:33389
> ldapRealm.userDnTemplate=uid={0},ou=people,dc=hadoop,dc=apache,dc=org
>
>
> # define filter: knoxResponseCookieFilter
> knoxResponseCookieFilter =
> org.apache.hadoop.gateway.filter.ResponseCookieFilter
> knoxResponseCookieFilter.enabled = true
> knoxResponseCookieFilter.filterHeaders = rememberMe, hadoop.auth.cookie
>
> # define filter: knoxPrincipalMapper
> knoxPrincipalMapper = org.apache.hadoop.gateway.filter.KnoxPrincipalMapper
> knoxPrincipalMapper.enabled = true
> knoxPrincipalMapper.userToUserMap = bob:guest, jon:bob
> knoxPrincipalMapper.userToGroupMap = *:users, bob:admin
>
> # define filter: knoxIPTracker
> knoxIPTracker =org.apache.hadoop.gateway.filter.KnoxIPTracker
> knoxIPTracker.enabled = true
>
> # define filter: knoxAclAuthzFilter
> knoxAclAuthzFilter = org.apache.hadoop.gateway.filter.KnoxAclAuthzFilter
> knoxAclAuthzFilter.enabled = true
> knoxAclAuthzFilter.globalGroupAclMode = OR
> knoxAclAuthzFilter.serviceGroupAclModeMap = /webhdfs/:OR, /templeton/:OR
> knoxAclAuthzFilter.serviceAclMap = /webhdfs/:user1 user2; users admin;
> 127.*.*.*, /templeton/:user11 user12; users admin
>
> # define filter: javaSubjectMapper
> javaSubjectMapper =
> org.apache.hadoop.gateway.filter.PostAuthenticationFilter
>
> # define filter: knoxIdentityAsserter
> knoxIdentityAsserter =
> org.apache.hadoop.gateway.filter.KnoxIdentityAsserter
> knoxIdentityAsserter.enabled = true
>
> # define filter: knoxUrlRewriter
> knoxUrlRewriter =
> org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter
>
> knoxUrlRewriter.requestUrlMap =
> /webhdfs/v1/:WEBHDFS/webhdfs/inbound/namenode/root,
> /webhdfs/v1/[^~]+:WEBHDFS/webhdfs/inbound/namenode/file,
> /webhdfs/v1/~:WEBHDFS/webhdfs/inbound/namenode/home,
> /webhdfs/v1/~/.*:WEBHDFS/webhdfs/inbound/namenode/home/file,
> /webhdfs/data/v1/.+:WEBHDFS/webhdfs/inbound/datanode
>
> knoxUrlRewriter.requestBodyMap = /oozie/:OOZIE/oozie/configuration,
> /oozie/v1/.*:OOZIE/oozie/configuration,
> /oozie/v2/.*:OOZIE/oozie/configuration
>
> knoxUrlRewriter.responseHeadersMap =
> /webhdfs/v1/[^~]*:WEBHDFS/webhdfs/outbound/namenode/headers,
> /webhdfs/v1/~/:WEBHDFS/webhdfs/outbound/namenode/headers,
> /hbase/:WEBHBASE/webhbase/headers/outbound,
> /hbase/[~/]*:WEBHBASE/webhbase/headers/outbound,
> /hbase/status/cluster:WEBHBASE/webhbase/status/outbound,
> /hbase/[^/]*/regions:WEBHBASE/webhbase/regions/outbound
>
> # define filter: knoxHttpDispatcher
> knoxHttpDispatcher = org.apache.hadoop.gateway.dispatch.HttpClientDispatch
> knoxHttpDispatcher.replayBufferSizeMap = webhdfs:4, templeton:8, oozie:4
>
> [urls]
> # you could choose to have a different chain of filter for different url
> patterns
> # so far Knox did not need it
> /** = knoxResponseCookieFilter, authcBasic, knoxPrincipalMapper,
> knoxIPTracker, knoxAclAuthzFilter, javaSubjectMapper, knoxIdentityAsserter,
> knoxUrlRewriter, knoxHttpDispatcher
>
> # end of shiro.ini
>
> I think we can rename shiro.ini as knox.ini to make it explicit this is
> more about knox configuration than shiro library configuration.
>
>
> We are using shiro config file mechanism as simple, lightweight depenency
> injection.
>
> This does not really tie us to using only Shiro authentication mechanism or
> authorization mechanisms.
>
> We have the choice of writing all our authentication or authorization in
> our own servlet filters or leverage filters from Shiro library, Realms from
> Shiro library or write your own Shiro filters or Shiro realms.
>
> In most of the cases, when we want to integrate with a new Hadoop back end
> service, all that we have to do is specify the path to a file having
> rewrite rules for the service. Rest of the things in knox.ini(=shiro.ini)
> would remain same.
>
> Please review and comment.
>
>
> Thanks
> Dilli
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>