You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@knox.apache.org by larry mccay <lm...@apache.org> on 2017/11/29 17:40:30 UTC
Re: Follow-up on KNOX-1119
Hi Andreas -
I am pulling this into the public lists for visibility and possibly
additional insights from our community.
This sounds like an interesting usecase.
I think that I need to understand the architectural aspects a bit more
however.
It seems to me that you have the following:
1. Custom services (REST I assume) that are higher level business type APIs
encapsulating the use of the lower level APIs through Knox
2. Users that are authenticating via KnoxSSO presumably through some custom
application (need more detail here)
3. You have considered accessing the custom services through Knox in order
to pass the knoxsso cookie to the backend service such that it could be
used to authenticate for access to other services back through Knox
I think that the above should work already but like I said, I would like to
see a more detailed depiction of this deployment so that I am sure that I
understand.
In my mind, there are a few possibilities here and you will need something
like the following inventory:
* Application that is being used as KnoxSSO token source. Meaning, without
a webapp for the browser to interact with the KnoxSSO cookie will not be
managed and presented by the browser.
* Application and Knox Gateway will both need to be in the same domain in
order for the cookie to be presented to the web service calls - which I
assume are coming from JS calls in the application
* Each custom web service will require a service definition in order for
Knox to be able to proxy access to them [1]
* The cookie should pass through to the proxied service by default along
with a user.name or doas query param that indicates the identity of the
gateway authenticated user
* A topology that hosts the cluster services that your web services will
need access to that is protected by the SSOCookieProvider [2]
The assumptions about the "application" above need to be verified and if
inaccurate, we need to understand who is managing the cookie. There may be
alternative approaches that we can use if this isn't quite right.
Thank you for your interest and contributions to Apache Knox!
Hope this is helpful.
--larry
1.
https://cwiki.apache.org/confluence/display/KNOX/2015/12/17/Adding+a+service+to+Apache+Knox
2.
http://knox.apache.org/books/knox-0-13-0/user-guide.html#SSO+Cookie+Provider
On Wed, Nov 29, 2017 at 8:50 AM, Hildebrandt, Prof. Dr. Andreas <
Andreas.Hildebrandt@uni-mainz.de> wrote:
> Dear Larry,
>
> thank you for the quick work on the patch! This is really great news! Now,
> we have a followup question.
>
> Our use cases require us to deploy several web services on the cluster.
> These services are supposed, according to the security concept, to go
> through knox to interact with the Hadoop cluster (e.g., to query hbase).
> The question we try to solve right now is how to authenticate the web
> services with knox. The security concept prevents us from using functional
> users and proxy user capabilities here.
>
> After long discussions with Alexandru and a lot of back and forth, we
> think that the best solution would be to put the web services behind knox,
> i.e., the user would connect to a knox endpoint which serves as a proxy for
> the web service. This seems simple enough. But we would need to forward, to
> the web service, a token that it can later use to authenticate against knox.
>
> As far as we can tell, when the user has authenticated with knox against
> our OpenID Connect provider, knox establishes a hadoop-jwt cookie which is
> later used for authentication. It seems to us that if we could forward that
> cookie from knox to the web service backend, the web service could later
> use that cookie to talk to, e.g., hbase through knox.
>
> Is that assumption correct? Is there any way we could achieve this in
> knox? If it requires patching the source code, could you give us a pointer
> what the correct place would be?
>
> I hope this explanation makes some sense!
>
> Best regards,
>
> Andreas
>
>
Re: Follow-up on KNOX-1119
Posted by larry mccay <lm...@apache.org>.
Okay - I have thrown together a quick sequence diagram and put it in a wiki
- since attachments don't work well on these lists:
https://cwiki.apache.org/confluence/display/KNOX/KnoxSSO+and+Custom+Application+Development?flashId=1313250692
Your interpretation of my description is correct.
It should just work if you line up everything properly. :)
One thing that is still not clear to me is how you are leveraging KnoxSSO
in your webapp today if you aren't proxying it.
Have you actually added native support for KnoxSSO in your application?
If you have, then we may need to require your webapp service definition
(which is needed to proxy it) to mandate the use of the Anonymous
authentication provider.
This means that the gateway will not authenticate requests for your app and
will delegate that responsibility to the app itself.
I would personally try and leverage the SSOCookieProvider since you are
going to be proxied anyway.
Native KnoxSSO integrations make it easier for you to determine the
effective user in the app though.
On Thu, Nov 30, 2017 at 4:36 AM, Hildebrandt, Prof. Dr. Andreas <
Andreas.Hildebrandt@uni-mainz.de> wrote:
> Dear Larry,
>
>
>
> thank you for your response!
>
> Our main use case is about interactive web services – web applications a
> user contacts to browse through the data on the data lake and perform some
> analysis. The back ends of these applications will have to talk to several
> services on the Hadoop cluster (such as HBase, or hdfs) on behalf of the
> end user connected to the web application. The common solution to the
> problem seems to be to have a functional user for the web application and
> then use proxy user functionality.
>
>
>
> What we would need instead is that the user first authenticates with
> KnoxSSO, connected to a federation server. This part already works for us.
> Now, we would like to put the web application behind knox (using it as a
> proxy). When the user contacts knox, knox should forward the token (the
> cookie) to the proxied web application. When the user then asks the web
> application to contact, say, hdfs, the back end should use that cookie to
> contact hdfs through knox.
>
>
>
> If I understood your email correctly, this should already work, right?
>
>
>
> - The token source would be the webapp the user interacts with.
> - We can easily put both in the same domain
> - We would have to build a service definition for each web app
>
>
>
> Did I understand it right that under these conditions, knox will
> automatically forward the cookie to the proxied service, and the proxied
> service can later use that cookie to contact knox to access, say, hdfs?
>
>
>
> This would be really great news and would greatly simplify our workflow! I
> also think that this is actually a quite common requirement.
>
>
>
> Best,
>
>
>
> Andreas
>
>
>
> *Von:* larry mccay [mailto:lmccay@apache.org]
> *Gesendet:* Mittwoch, 29. November 2017 18:41
> *An:* Hildebrandt, Prof. Dr. Andreas <An...@uni-mainz.de>;
> dev@knox.apache.org; user@knox.apache.org
> *Cc:* Alexandru Anghel <aa...@hortonworks.com>; Kartik Ramalingam <
> kramalingam@hortonworks.com>; Rolf Schulz <rs...@gnsec.com>; Dagmar Melanie
> Doll <da...@basf.com>
> *Betreff:* Re: Follow-up on KNOX-1119
>
>
>
> Hi Andreas -
>
>
>
> I am pulling this into the public lists for visibility and possibly
> additional insights from our community.
>
>
>
> This sounds like an interesting usecase.
>
> I think that I need to understand the architectural aspects a bit more
> however.
>
>
>
> It seems to me that you have the following:
>
>
>
> 1. Custom services (REST I assume) that are higher level business type
> APIs encapsulating the use of the lower level APIs through Knox
>
> 2. Users that are authenticating via KnoxSSO presumably through some
> custom application (need more detail here)
>
> 3. You have considered accessing the custom services through Knox in order
> to pass the knoxsso cookie to the backend service such that it could be
> used to authenticate for access to other services back through Knox
>
>
>
> I think that the above should work already but like I said, I would like
> to see a more detailed depiction of this deployment so that I am sure that
> I understand.
>
>
>
> In my mind, there are a few possibilities here and you will need something
> like the following inventory:
>
>
>
> * Application that is being used as KnoxSSO token source. Meaning, without
> a webapp for the browser to interact with the KnoxSSO cookie will not be
> managed and presented by the browser.
>
> * Application and Knox Gateway will both need to be in the same domain in
> order for the cookie to be presented to the web service calls - which I
> assume are coming from JS calls in the application
>
> * Each custom web service will require a service definition in order for
> Knox to be able to proxy access to them [1]
>
> * The cookie should pass through to the proxied service by default along
> with a user.name or doas query param that indicates the identity of the
> gateway authenticated user
>
> * A topology that hosts the cluster services that your web services will
> need access to that is protected by the SSOCookieProvider [2]
>
>
>
> The assumptions about the "application" above need to be verified and if
> inaccurate, we need to understand who is managing the cookie. There may be
> alternative approaches that we can use if this isn't quite right.
>
>
>
> Thank you for your interest and contributions to Apache Knox!
>
>
>
> Hope this is helpful.
>
>
>
> --larry
>
>
>
> 1. https://cwiki.apache.org/confluence/display/KNOX/2015/
> 12/17/Adding+a+service+to+Apache+Knox
>
> 2. http://knox.apache.org/books/knox-0-13-0/user-guide.
> html#SSO+Cookie+Provider
>
>
>
>
>
> On Wed, Nov 29, 2017 at 8:50 AM, Hildebrandt, Prof. Dr. Andreas <
> Andreas.Hildebrandt@uni-mainz.de> wrote:
>
> Dear Larry,
>
>
>
> thank you for the quick work on the patch! This is really great news! Now,
> we have a followup question.
>
>
>
> Our use cases require us to deploy several web services on the cluster.
> These services are supposed, according to the security concept, to go
> through knox to interact with the Hadoop cluster (e.g., to query hbase).
> The question we try to solve right now is how to authenticate the web
> services with knox. The security concept prevents us from using functional
> users and proxy user capabilities here.
>
>
>
> After long discussions with Alexandru and a lot of back and forth, we
> think that the best solution would be to put the web services behind knox,
> i.e., the user would connect to a knox endpoint which serves as a proxy for
> the web service. This seems simple enough. But we would need to forward, to
> the web service, a token that it can later use to authenticate against knox.
>
>
>
> As far as we can tell, when the user has authenticated with knox against
> our OpenID Connect provider, knox establishes a hadoop-jwt cookie which is
> later used for authentication. It seems to us that if we could forward that
> cookie from knox to the web service backend, the web service could later
> use that cookie to talk to, e.g., hbase through knox.
>
>
>
> Is that assumption correct? Is there any way we could achieve this in
> knox? If it requires patching the source code, could you give us a pointer
> what the correct place would be?
>
>
>
> I hope this explanation makes some sense!
>
>
>
> Best regards,
>
>
>
> Andreas
>
>
>
>
>
Re: Follow-up on KNOX-1119
Posted by larry mccay <lm...@apache.org>.
Okay - I have thrown together a quick sequence diagram and put it in a wiki
- since attachments don't work well on these lists:
https://cwiki.apache.org/confluence/display/KNOX/KnoxSSO+and+Custom+Application+Development?flashId=1313250692
Your interpretation of my description is correct.
It should just work if you line up everything properly. :)
One thing that is still not clear to me is how you are leveraging KnoxSSO
in your webapp today if you aren't proxying it.
Have you actually added native support for KnoxSSO in your application?
If you have, then we may need to require your webapp service definition
(which is needed to proxy it) to mandate the use of the Anonymous
authentication provider.
This means that the gateway will not authenticate requests for your app and
will delegate that responsibility to the app itself.
I would personally try and leverage the SSOCookieProvider since you are
going to be proxied anyway.
Native KnoxSSO integrations make it easier for you to determine the
effective user in the app though.
On Thu, Nov 30, 2017 at 4:36 AM, Hildebrandt, Prof. Dr. Andreas <
Andreas.Hildebrandt@uni-mainz.de> wrote:
> Dear Larry,
>
>
>
> thank you for your response!
>
> Our main use case is about interactive web services – web applications a
> user contacts to browse through the data on the data lake and perform some
> analysis. The back ends of these applications will have to talk to several
> services on the Hadoop cluster (such as HBase, or hdfs) on behalf of the
> end user connected to the web application. The common solution to the
> problem seems to be to have a functional user for the web application and
> then use proxy user functionality.
>
>
>
> What we would need instead is that the user first authenticates with
> KnoxSSO, connected to a federation server. This part already works for us.
> Now, we would like to put the web application behind knox (using it as a
> proxy). When the user contacts knox, knox should forward the token (the
> cookie) to the proxied web application. When the user then asks the web
> application to contact, say, hdfs, the back end should use that cookie to
> contact hdfs through knox.
>
>
>
> If I understood your email correctly, this should already work, right?
>
>
>
> - The token source would be the webapp the user interacts with.
> - We can easily put both in the same domain
> - We would have to build a service definition for each web app
>
>
>
> Did I understand it right that under these conditions, knox will
> automatically forward the cookie to the proxied service, and the proxied
> service can later use that cookie to contact knox to access, say, hdfs?
>
>
>
> This would be really great news and would greatly simplify our workflow! I
> also think that this is actually a quite common requirement.
>
>
>
> Best,
>
>
>
> Andreas
>
>
>
> *Von:* larry mccay [mailto:lmccay@apache.org]
> *Gesendet:* Mittwoch, 29. November 2017 18:41
> *An:* Hildebrandt, Prof. Dr. Andreas <An...@uni-mainz.de>;
> dev@knox.apache.org; user@knox.apache.org
> *Cc:* Alexandru Anghel <aa...@hortonworks.com>; Kartik Ramalingam <
> kramalingam@hortonworks.com>; Rolf Schulz <rs...@gnsec.com>; Dagmar Melanie
> Doll <da...@basf.com>
> *Betreff:* Re: Follow-up on KNOX-1119
>
>
>
> Hi Andreas -
>
>
>
> I am pulling this into the public lists for visibility and possibly
> additional insights from our community.
>
>
>
> This sounds like an interesting usecase.
>
> I think that I need to understand the architectural aspects a bit more
> however.
>
>
>
> It seems to me that you have the following:
>
>
>
> 1. Custom services (REST I assume) that are higher level business type
> APIs encapsulating the use of the lower level APIs through Knox
>
> 2. Users that are authenticating via KnoxSSO presumably through some
> custom application (need more detail here)
>
> 3. You have considered accessing the custom services through Knox in order
> to pass the knoxsso cookie to the backend service such that it could be
> used to authenticate for access to other services back through Knox
>
>
>
> I think that the above should work already but like I said, I would like
> to see a more detailed depiction of this deployment so that I am sure that
> I understand.
>
>
>
> In my mind, there are a few possibilities here and you will need something
> like the following inventory:
>
>
>
> * Application that is being used as KnoxSSO token source. Meaning, without
> a webapp for the browser to interact with the KnoxSSO cookie will not be
> managed and presented by the browser.
>
> * Application and Knox Gateway will both need to be in the same domain in
> order for the cookie to be presented to the web service calls - which I
> assume are coming from JS calls in the application
>
> * Each custom web service will require a service definition in order for
> Knox to be able to proxy access to them [1]
>
> * The cookie should pass through to the proxied service by default along
> with a user.name or doas query param that indicates the identity of the
> gateway authenticated user
>
> * A topology that hosts the cluster services that your web services will
> need access to that is protected by the SSOCookieProvider [2]
>
>
>
> The assumptions about the "application" above need to be verified and if
> inaccurate, we need to understand who is managing the cookie. There may be
> alternative approaches that we can use if this isn't quite right.
>
>
>
> Thank you for your interest and contributions to Apache Knox!
>
>
>
> Hope this is helpful.
>
>
>
> --larry
>
>
>
> 1. https://cwiki.apache.org/confluence/display/KNOX/2015/
> 12/17/Adding+a+service+to+Apache+Knox
>
> 2. http://knox.apache.org/books/knox-0-13-0/user-guide.
> html#SSO+Cookie+Provider
>
>
>
>
>
> On Wed, Nov 29, 2017 at 8:50 AM, Hildebrandt, Prof. Dr. Andreas <
> Andreas.Hildebrandt@uni-mainz.de> wrote:
>
> Dear Larry,
>
>
>
> thank you for the quick work on the patch! This is really great news! Now,
> we have a followup question.
>
>
>
> Our use cases require us to deploy several web services on the cluster.
> These services are supposed, according to the security concept, to go
> through knox to interact with the Hadoop cluster (e.g., to query hbase).
> The question we try to solve right now is how to authenticate the web
> services with knox. The security concept prevents us from using functional
> users and proxy user capabilities here.
>
>
>
> After long discussions with Alexandru and a lot of back and forth, we
> think that the best solution would be to put the web services behind knox,
> i.e., the user would connect to a knox endpoint which serves as a proxy for
> the web service. This seems simple enough. But we would need to forward, to
> the web service, a token that it can later use to authenticate against knox.
>
>
>
> As far as we can tell, when the user has authenticated with knox against
> our OpenID Connect provider, knox establishes a hadoop-jwt cookie which is
> later used for authentication. It seems to us that if we could forward that
> cookie from knox to the web service backend, the web service could later
> use that cookie to talk to, e.g., hbase through knox.
>
>
>
> Is that assumption correct? Is there any way we could achieve this in
> knox? If it requires patching the source code, could you give us a pointer
> what the correct place would be?
>
>
>
> I hope this explanation makes some sense!
>
>
>
> Best regards,
>
>
>
> Andreas
>
>
>
>
>
AW: Follow-up on KNOX-1119
Posted by "Hildebrandt, Prof. Dr. Andreas" <An...@uni-mainz.de>.
Dear Larry,
thank you for your response!
Our main use case is about interactive web services – web applications a user contacts to browse through the data on the data lake and perform some analysis. The back ends of these applications will have to talk to several services on the Hadoop cluster (such as HBase, or hdfs) on behalf of the end user connected to the web application. The common solution to the problem seems to be to have a functional user for the web application and then use proxy user functionality.
What we would need instead is that the user first authenticates with KnoxSSO, connected to a federation server. This part already works for us. Now, we would like to put the web application behind knox (using it as a proxy). When the user contacts knox, knox should forward the token (the cookie) to the proxied web application. When the user then asks the web application to contact, say, hdfs, the back end should use that cookie to contact hdfs through knox.
If I understood your email correctly, this should already work, right?
* The token source would be the webapp the user interacts with.
* We can easily put both in the same domain
* We would have to build a service definition for each web app
Did I understand it right that under these conditions, knox will automatically forward the cookie to the proxied service, and the proxied service can later use that cookie to contact knox to access, say, hdfs?
This would be really great news and would greatly simplify our workflow! I also think that this is actually a quite common requirement.
Best,
Andreas
Von: larry mccay [mailto:lmccay@apache.org]
Gesendet: Mittwoch, 29. November 2017 18:41
An: Hildebrandt, Prof. Dr. Andreas <An...@uni-mainz.de>; dev@knox.apache.org; user@knox.apache.org
Cc: Alexandru Anghel <aa...@hortonworks.com>; Kartik Ramalingam <kr...@hortonworks.com>; Rolf Schulz <rs...@gnsec.com>; Dagmar Melanie Doll <da...@basf.com>
Betreff: Re: Follow-up on KNOX-1119
Hi Andreas -
I am pulling this into the public lists for visibility and possibly additional insights from our community.
This sounds like an interesting usecase.
I think that I need to understand the architectural aspects a bit more however.
It seems to me that you have the following:
1. Custom services (REST I assume) that are higher level business type APIs encapsulating the use of the lower level APIs through Knox
2. Users that are authenticating via KnoxSSO presumably through some custom application (need more detail here)
3. You have considered accessing the custom services through Knox in order to pass the knoxsso cookie to the backend service such that it could be used to authenticate for access to other services back through Knox
I think that the above should work already but like I said, I would like to see a more detailed depiction of this deployment so that I am sure that I understand.
In my mind, there are a few possibilities here and you will need something like the following inventory:
* Application that is being used as KnoxSSO token source. Meaning, without a webapp for the browser to interact with the KnoxSSO cookie will not be managed and presented by the browser.
* Application and Knox Gateway will both need to be in the same domain in order for the cookie to be presented to the web service calls - which I assume are coming from JS calls in the application
* Each custom web service will require a service definition in order for Knox to be able to proxy access to them [1]
* The cookie should pass through to the proxied service by default along with a user.name<http://user.name> or doas query param that indicates the identity of the gateway authenticated user
* A topology that hosts the cluster services that your web services will need access to that is protected by the SSOCookieProvider [2]
The assumptions about the "application" above need to be verified and if inaccurate, we need to understand who is managing the cookie. There may be alternative approaches that we can use if this isn't quite right.
Thank you for your interest and contributions to Apache Knox!
Hope this is helpful.
--larry
1. https://cwiki.apache.org/confluence/display/KNOX/2015/12/17/Adding+a+service+to+Apache+Knox
2. http://knox.apache.org/books/knox-0-13-0/user-guide.html#SSO+Cookie+Provider
On Wed, Nov 29, 2017 at 8:50 AM, Hildebrandt, Prof. Dr. Andreas <An...@uni-mainz.de>> wrote:
Dear Larry,
thank you for the quick work on the patch! This is really great news! Now, we have a followup question.
Our use cases require us to deploy several web services on the cluster. These services are supposed, according to the security concept, to go through knox to interact with the Hadoop cluster (e.g., to query hbase). The question we try to solve right now is how to authenticate the web services with knox. The security concept prevents us from using functional users and proxy user capabilities here.
After long discussions with Alexandru and a lot of back and forth, we think that the best solution would be to put the web services behind knox, i.e., the user would connect to a knox endpoint which serves as a proxy for the web service. This seems simple enough. But we would need to forward, to the web service, a token that it can later use to authenticate against knox.
As far as we can tell, when the user has authenticated with knox against our OpenID Connect provider, knox establishes a hadoop-jwt cookie which is later used for authentication. It seems to us that if we could forward that cookie from knox to the web service backend, the web service could later use that cookie to talk to, e.g., hbase through knox.
Is that assumption correct? Is there any way we could achieve this in knox? If it requires patching the source code, could you give us a pointer what the correct place would be?
I hope this explanation makes some sense!
Best regards,
Andreas