You are viewing a plain text version of this content. The canonical link for it is here.
Posted to custos@airavata.apache.org by "Pierce, Marlon" <ma...@iu.edu> on 2019/08/20 16:24:58 UTC
Re: [External] Re: Updates and Discussions
I also put together some slides that attempted to synthesize these two sources for our class at IU: https://github.com/airavata-courses/airavata-courses.github.io/blob/master/slides/Fall2018-OAuth2-OIDC.pptx.
Marlon
On 8/20/19, 12:20 PM, "Suresh Marru" <sm...@apache.org> wrote:
This message was sent from a non-IU address. Please exercise caution when clicking links or opening attachments from external sources.
-------
Hi Aarushi,
If you ave questions on authentication grant flows and its application to gateways, this paper has some description - https://scholarworks.iu.edu/dspace/bitstream/handle/2022/21092/airavata-security-escience16.pdf <https://scholarworks.iu.edu/dspace/bitstream/handle/2022/21092/airavata-security-escience16.pdf>
Also OAuth2 spec describes the flows well - https://tools.ietf.org/html/rfc6749#page-23 <https://tools.ietf.org/html/rfc6749#page-23>
Suresh
> On Aug 20, 2019, at 10:27 AM, Suresh Marru <sm...@apache.org> wrote:
>
> Hi Aarushi,
>
> Thank you for the detailed update. Looks like Marcus’s response addressed your queries, I will echo them:
>
> I agree with Marcus the sharing and group services are inherently coupled but as you said it will helpful to facilitate use of group capabilities without necessarily having to use sharing. So + 1 for keeping them in one service but exposing group only API’s to go along with User Profile API’s.
>
> + 1 for OAuth2 Client Credentials as an authentication mechanism between Gateways and Custos. Since Gateways are hosted on a secured services, this might work well.
>
> + 1 for Airavata proxy API’s for Custos functionality. This might help preserve usability of Airavata API’s while allowing independent use of Custos Services.
>
> Suresh
>
>> On Aug 19, 2019, at 5:09 PM, Christie, Marcus Aaron <ma...@iu.edu> wrote:
>>
>> Thanks for the update Aarushi. Replies inline:
>>
>> On 8/18/19, 6:25 PM, "Bisht, Aarushi" <ab...@iu.edu> wrote:
>>
>> 1. ??Group Service Migration:??? This service is tightly coupled with the airavata sharing registry service. Since sharing service will also be migrated to custos, I could defer the migration of group service until the migration of sharing service while keeping the dependency between these 2, or, I could decouple this dependency and make group service an independent service. We might need independent group service if we are allowing other Gateways to use only Custos authentication and profile service while implementing their own sharing service. What are our plans regarding this?
>>
>>
>> The GroupManagerService essentially just adds authentication on the group functionality in the sharing service. I think they are inherently coupled. I'm assuming Custos will need some similar public API to its functionality.
>>
>>
>> 2. Integration of Custos with Airavata: There are couple of discussions that I wanted to bring up:
>>
>> * ? Authentication between Gateways and Custos : As of now, I have not implemented any authentication/authorization between these 2 services. Do we need this or would we rely on network and firewall based limited access to custos servers? If we do need it, I would appreciate suggestions on type of authentication to use.
>>
>>
>> Since Custos will be potentially consumed by multiple non-Airavata gateways, I think it needs to authenticate/authorize calls. For service-to-service type calls my initial thought is to use OAuth2 client credentials.
>>
>>
>> * ??Airavata proxying Custos services: Since the portals using Airavata should not be impacted with the migration of some of Airavata services, the Airavata APIs will essentially be proxying Custos services in some cases. For example the Airavata APIs for user creation,deletion, updation will remain the same but most of the business logic and database will be in Custos. Airavata will only keep user/tenant information specific to itself. So essentially, Airavata is just proxying Custos. This is true for most of the APIs which rely on Custos.
>>
>>
>> I think proxying from the Airavata services to Custos sounds like a good idea.
>
Re: [External] Re: Updates and Discussions
Posted by Thejaka Amila J Kanewala <th...@gmail.com>.
Hi Aarushi,
Some comments inline.
Thank you
Thejaka
On Mon, Aug 26, 2019 at 9:02 AM Bisht, Aarushi <ab...@iu.edu> wrote:
>
>
> Hi,
>
>
> Thank you Thejaka for pointing out the issue. In
> UserProfileServiceHandler.java:95, a new UserProfileServiceException is
> created and original exception message is included in the message of the
> new exception. Please let me know if this is not the correct approach.
>
>
> For the test cases to run successfully we need docker setup first. The
> instructions on how to setup the services locally on intellij are in this <
> https://github.com/apache/airavata-custos/pull/4> PR. The PR has not been
> merged to the develop branch yet but you can find the readme file there. I
> have also included some samples in the PR that create service clients and
> tests them. These are thrift clients as of now but will be changed to REST
> in the future. The samples can be found in ide-integration-> samples
> module. Please let me know if anyone faces any difficulty setting up the
> environment.
>
I hope this approach will be changed in future -- As far as engineering is
concerned, we need a way to build the project without depending upon a
specific IDE. In summary, we should be able to build a java project with
the following two commands:
> git clone <URI> project
> cd project
> mvn clean install
Any pre-setup other than the above is an overhead. If we have to do some
pre-setup for tests, better to do them using maven plugins or scripts that
execute at the maven command.
Further, I do no clearly understand why you need a docker container -- is
it to start Keycloak? (I am unable to locate CustosAPIServerStarter class
in the repo as well. Is it in a different pull request ?)
>
> PLANS
>
> Here are the plans and action items that I will be working on in next few
> weeks:
>
> Migration of APIs from thrift to REST: As discussed in the last meeting, I
> will be changing all custos public APIs to REST. Since the already
> developed custos credential-store uses Spring framework, I am also planning
> to use Spring Boot 2.1.6.RELEASE for developing the APIs to maintain
> consistency across the project.
>
>
> Migration of users group and sharing service: I will be implementing REST
> APIs for these services.
>
>
> Authentication between gateways and Custos: I will be implementing OAuth2
> Client Credential grant type using keycloak.
>
I am not sure OAuth2 client credential is the right choice in this case.
OAuth2 client credential assures secure passage for the communication
between the gateway and Custos; which means we still need to have manual
calls to perform authentication. Further gateway still needs to handle the
authentication with user browser (e.g., using basic auth). For a start, we
can do this but in future, we need delegated authorization (authorization
grant) as users need to use their google accounts, CILogon etc., to login
to their gateways.
Just to make sure I understand your approach, please give an example of how
authorization works with client credential grant (take a simple example
like submitting a job).
>
>
> I am currently working on migration of user group and sharing service. We
> had a doubt regarding this which we were not able discuss during the
> meeting last Tuesday. Since the use of user groups is limited to sharing
> resources in Airavata, I wanted to ask if there is any other use case in
> Galaxy which requires user groups for any other purpose other than managing
> the sharing of resources.
>
>
> Thanks & Regards,
>
> Aarushi Bisht
>
>
> ________________________________
> From: Thejaka Amila J Kanewala <th...@gmail.com>
> Sent: Monday, August 26, 2019 2:01 AM
> To: custos@airavata.apache.org
> Subject: Re: [External] Re: Updates and Discussions
>
> Hello Aarushi,
>
> Sorry for the late reply.
> It seems you already got answers to your questions.
>
> As mentioned by others +1 for OAuth and that was our initial plan as well
> (see kick-off slides).
>
> In detail: Our initial idea was to use OAuth browser-based profile to
> perform authorization and authentication (OpenID mode). A user tries to
> perform an action on the gateway UI, if the user is not already authorized
> to perform the action then, the user is re-directed to an authorization
> server (in this case Custos). Custos perform the authorization and
> redirects the user browser to the callback (Completes the OAuth flow). I
> believe we need to modify Airavata client API appropriately (not quite sure
> how, though)
>
> In addition, I found minor engineering issues in the develop branch.
> - There are some test failures in TestUserProfileServiceHandler due to
> connection failures. Seems like we are swallowing the original exception in
> UserProfileServiceHandler.java:95 by instantiating a new exception without
> referring the original exception. It's good to have a reference to original
> exception.
>
> Good progress.
>
> Thanks
> -Thejaka
>
> On Tue, Aug 20, 2019 at 12:25 PM Pierce, Marlon <marpierc@iu.edu<mailto:
> marpierc@iu.edu>> wrote:
> I also put together some slides that attempted to synthesize these two
> sources for our class at IU:
> https://github.com/airavata-courses/airavata-courses.github.io/blob/master/slides/Fall2018-OAuth2-OIDC.pptx
> .
>
> Marlon
>
>
> On 8/20/19, 12:20 PM, "Suresh Marru" <smarru@apache.org<mailto:
> smarru@apache.org>> wrote:
>
> This message was sent from a non-IU address. Please exercise caution
> when clicking links or opening attachments from external sources.
> -------
>
> Hi Aarushi,
>
> If you ave questions on authentication grant flows and its application
> to gateways, this paper has some description -
> https://scholarworks.iu.edu/dspace/bitstream/handle/2022/21092/airavata-security-escience16.pdf
> <
> https://scholarworks.iu.edu/dspace/bitstream/handle/2022/21092/airavata-security-escience16.pdf
> >
>
> Also OAuth2 spec describes the flows well -
> https://tools.ietf.org/html/rfc6749#page-23 <
> https://tools.ietf.org/html/rfc6749#page-23>
>
> Suresh
>
> > On Aug 20, 2019, at 10:27 AM, Suresh Marru <smarru@apache.org
> <ma...@apache.org>> wrote:
> >
> > Hi Aarushi,
> >
> > Thank you for the detailed update. Looks like Marcus’s response
> addressed your queries, I will echo them:
> >
> > I agree with Marcus the sharing and group services are inherently
> coupled but as you said it will helpful to facilitate use of group
> capabilities without necessarily having to use sharing. So + 1 for keeping
> them in one service but exposing group only API’s to go along with User
> Profile API’s.
> >
> > + 1 for OAuth2 Client Credentials as an authentication mechanism
> between Gateways and Custos. Since Gateways are hosted on a secured
> services, this might work well.
> >
> > + 1 for Airavata proxy API’s for Custos functionality. This might
> help preserve usability of Airavata API’s while allowing independent use of
> Custos Services.
> >
> > Suresh
> >
> >> On Aug 19, 2019, at 5:09 PM, Christie, Marcus Aaron <
> machrist@iu.edu<ma...@iu.edu>> wrote:
> >>
> >> Thanks for the update Aarushi. Replies inline:
> >>
> >> On 8/18/19, 6:25 PM, "Bisht, Aarushi" <abisht@iu.edu<mailto:
> abisht@iu.edu>> wrote:
> >>
> >> 1. ??Group Service Migration:??? This service is tightly
> coupled with the airavata sharing registry service. Since sharing service
> will also be migrated to custos, I could defer the migration of group
> service until the migration of sharing service while keeping the dependency
> between these 2, or, I could decouple this dependency and make group
> service an independent service. We might need independent group service if
> we are allowing other Gateways to use only Custos authentication and
> profile service while implementing their own sharing service. What are our
> plans regarding this?
> >>
> >>
> >> The GroupManagerService essentially just adds authentication on the
> group functionality in the sharing service. I think they are inherently
> coupled. I'm assuming Custos will need some similar public API to its
> functionality.
> >>
> >>
> >> 2. Integration of Custos with Airavata: There are couple of
> discussions that I wanted to bring up:
> >>
> >> * ? Authentication between Gateways and Custos : As of now, I
> have not implemented any authentication/authorization between these 2
> services. Do we need this or would we rely on network and firewall based
> limited access to custos servers? If we do need it, I would appreciate
> suggestions on type of authentication to use.
> >>
> >>
> >> Since Custos will be potentially consumed by multiple non-Airavata
> gateways, I think it needs to authenticate/authorize calls. For
> service-to-service type calls my initial thought is to use OAuth2 client
> credentials.
> >>
> >>
> >> * ??Airavata proxying Custos services: Since the portals
> using Airavata should not be impacted with the migration of some of
> Airavata services, the Airavata APIs will essentially be proxying Custos
> services in some cases. For example the Airavata APIs for user
> creation,deletion, updation will remain the same but most of the business
> logic and database will be in Custos. Airavata will only keep user/tenant
> information specific to itself. So essentially, Airavata is just proxying
> Custos. This is true for most of the APIs which rely on Custos.
> >>
> >>
> >> I think proxying from the Airavata services to Custos sounds like a
> good idea.
> >
>
>
>
>
> --
> Best Regards,
> Thejaka Amila Kanewala, PhD
> https://github.com/thejkane/agm
>
--
Best Regards,
Thejaka Amila Kanewala, PhD
https://github.com/thejkane/agm
Re: [External] Re: Updates and Discussions
Posted by "Bisht, Aarushi" <ab...@iu.edu>.
Hi,
Thank you Thejaka for pointing out the issue. In UserProfileServiceHandler.java:95, a new UserProfileServiceException is created and original exception message is included in the message of the new exception. Please let me know if this is not the correct approach.
For the test cases to run successfully we need docker setup first. The instructions on how to setup the services locally on intellij are in this <https://github.com/apache/airavata-custos/pull/4> PR. The PR has not been merged to the develop branch yet but you can find the readme file there. I have also included some samples in the PR that create service clients and tests them. These are thrift clients as of now but will be changed to REST in the future. The samples can be found in ide-integration-> samples module. Please let me know if anyone faces any difficulty setting up the environment.
PLANS
Here are the plans and action items that I will be working on in next few weeks:
Migration of APIs from thrift to REST: As discussed in the last meeting, I will be changing all custos public APIs to REST. Since the already developed custos credential-store uses Spring framework, I am also planning to use Spring Boot 2.1.6.RELEASE for developing the APIs to maintain consistency across the project.
Migration of users group and sharing service: I will be implementing REST APIs for these services.
Authentication between gateways and Custos: I will be implementing OAuth2 Client Credential grant type using keycloak.
I am currently working on migration of user group and sharing service. We had a doubt regarding this which we were not able discuss during the meeting last Tuesday. Since the use of user groups is limited to sharing resources in Airavata, I wanted to ask if there is any other use case in Galaxy which requires user groups for any other purpose other than managing the sharing of resources.
Thanks & Regards,
Aarushi Bisht
________________________________
From: Thejaka Amila J Kanewala <th...@gmail.com>
Sent: Monday, August 26, 2019 2:01 AM
To: custos@airavata.apache.org
Subject: Re: [External] Re: Updates and Discussions
Hello Aarushi,
Sorry for the late reply.
It seems you already got answers to your questions.
As mentioned by others +1 for OAuth and that was our initial plan as well (see kick-off slides).
In detail: Our initial idea was to use OAuth browser-based profile to perform authorization and authentication (OpenID mode). A user tries to perform an action on the gateway UI, if the user is not already authorized to perform the action then, the user is re-directed to an authorization server (in this case Custos). Custos perform the authorization and redirects the user browser to the callback (Completes the OAuth flow). I believe we need to modify Airavata client API appropriately (not quite sure how, though)
In addition, I found minor engineering issues in the develop branch.
- There are some test failures in TestUserProfileServiceHandler due to connection failures. Seems like we are swallowing the original exception in UserProfileServiceHandler.java:95 by instantiating a new exception without referring the original exception. It's good to have a reference to original exception.
Good progress.
Thanks
-Thejaka
On Tue, Aug 20, 2019 at 12:25 PM Pierce, Marlon <ma...@iu.edu>> wrote:
I also put together some slides that attempted to synthesize these two sources for our class at IU: https://github.com/airavata-courses/airavata-courses.github.io/blob/master/slides/Fall2018-OAuth2-OIDC.pptx.
Marlon
On 8/20/19, 12:20 PM, "Suresh Marru" <sm...@apache.org>> wrote:
This message was sent from a non-IU address. Please exercise caution when clicking links or opening attachments from external sources.
-------
Hi Aarushi,
If you ave questions on authentication grant flows and its application to gateways, this paper has some description - https://scholarworks.iu.edu/dspace/bitstream/handle/2022/21092/airavata-security-escience16.pdf <https://scholarworks.iu.edu/dspace/bitstream/handle/2022/21092/airavata-security-escience16.pdf>
Also OAuth2 spec describes the flows well - https://tools.ietf.org/html/rfc6749#page-23 <https://tools.ietf.org/html/rfc6749#page-23>
Suresh
> On Aug 20, 2019, at 10:27 AM, Suresh Marru <sm...@apache.org>> wrote:
>
> Hi Aarushi,
>
> Thank you for the detailed update. Looks like Marcus’s response addressed your queries, I will echo them:
>
> I agree with Marcus the sharing and group services are inherently coupled but as you said it will helpful to facilitate use of group capabilities without necessarily having to use sharing. So + 1 for keeping them in one service but exposing group only API’s to go along with User Profile API’s.
>
> + 1 for OAuth2 Client Credentials as an authentication mechanism between Gateways and Custos. Since Gateways are hosted on a secured services, this might work well.
>
> + 1 for Airavata proxy API’s for Custos functionality. This might help preserve usability of Airavata API’s while allowing independent use of Custos Services.
>
> Suresh
>
>> On Aug 19, 2019, at 5:09 PM, Christie, Marcus Aaron <ma...@iu.edu>> wrote:
>>
>> Thanks for the update Aarushi. Replies inline:
>>
>> On 8/18/19, 6:25 PM, "Bisht, Aarushi" <ab...@iu.edu>> wrote:
>>
>> 1. ??Group Service Migration:??? This service is tightly coupled with the airavata sharing registry service. Since sharing service will also be migrated to custos, I could defer the migration of group service until the migration of sharing service while keeping the dependency between these 2, or, I could decouple this dependency and make group service an independent service. We might need independent group service if we are allowing other Gateways to use only Custos authentication and profile service while implementing their own sharing service. What are our plans regarding this?
>>
>>
>> The GroupManagerService essentially just adds authentication on the group functionality in the sharing service. I think they are inherently coupled. I'm assuming Custos will need some similar public API to its functionality.
>>
>>
>> 2. Integration of Custos with Airavata: There are couple of discussions that I wanted to bring up:
>>
>> * ? Authentication between Gateways and Custos : As of now, I have not implemented any authentication/authorization between these 2 services. Do we need this or would we rely on network and firewall based limited access to custos servers? If we do need it, I would appreciate suggestions on type of authentication to use.
>>
>>
>> Since Custos will be potentially consumed by multiple non-Airavata gateways, I think it needs to authenticate/authorize calls. For service-to-service type calls my initial thought is to use OAuth2 client credentials.
>>
>>
>> * ??Airavata proxying Custos services: Since the portals using Airavata should not be impacted with the migration of some of Airavata services, the Airavata APIs will essentially be proxying Custos services in some cases. For example the Airavata APIs for user creation,deletion, updation will remain the same but most of the business logic and database will be in Custos. Airavata will only keep user/tenant information specific to itself. So essentially, Airavata is just proxying Custos. This is true for most of the APIs which rely on Custos.
>>
>>
>> I think proxying from the Airavata services to Custos sounds like a good idea.
>
--
Best Regards,
Thejaka Amila Kanewala, PhD
https://github.com/thejkane/agm
Re: [External] Re: Updates and Discussions
Posted by Thejaka Amila J Kanewala <th...@gmail.com>.
Hello Aarushi,
Sorry for the late reply.
It seems you already got answers to your questions.
As mentioned by others +1 for OAuth and that was our initial plan as well
(see kick-off slides).
In detail: Our initial idea was to use OAuth browser-based profile to
perform authorization and authentication (OpenID mode). A user tries to
perform an action on the gateway UI, if the user is not already authorized
to perform the action then, the user is re-directed to an authorization
server (in this case Custos). Custos perform the authorization and
redirects the user browser to the callback (Completes the OAuth flow). I
believe we need to modify Airavata client API appropriately (not quite sure
how, though)
In addition, I found minor engineering issues in the develop branch.
- There are some test failures in TestUserProfileServiceHandler due to
connection failures. Seems like we are swallowing the original exception
in UserProfileServiceHandler.java:95 by instantiating a new exception
without referring the original exception. It's good to have a reference to
original exception.
Good progress.
Thanks
-Thejaka
On Tue, Aug 20, 2019 at 12:25 PM Pierce, Marlon <ma...@iu.edu> wrote:
> I also put together some slides that attempted to synthesize these two
> sources for our class at IU:
> https://github.com/airavata-courses/airavata-courses.github.io/blob/master/slides/Fall2018-OAuth2-OIDC.pptx
> .
>
> Marlon
>
>
> On 8/20/19, 12:20 PM, "Suresh Marru" <sm...@apache.org> wrote:
>
> This message was sent from a non-IU address. Please exercise caution
> when clicking links or opening attachments from external sources.
> -------
>
> Hi Aarushi,
>
> If you ave questions on authentication grant flows and its application
> to gateways, this paper has some description -
> https://scholarworks.iu.edu/dspace/bitstream/handle/2022/21092/airavata-security-escience16.pdf
> <
> https://scholarworks.iu.edu/dspace/bitstream/handle/2022/21092/airavata-security-escience16.pdf
> >
>
> Also OAuth2 spec describes the flows well -
> https://tools.ietf.org/html/rfc6749#page-23 <
> https://tools.ietf.org/html/rfc6749#page-23>
>
> Suresh
>
> > On Aug 20, 2019, at 10:27 AM, Suresh Marru <sm...@apache.org>
> wrote:
> >
> > Hi Aarushi,
> >
> > Thank you for the detailed update. Looks like Marcus’s response
> addressed your queries, I will echo them:
> >
> > I agree with Marcus the sharing and group services are inherently
> coupled but as you said it will helpful to facilitate use of group
> capabilities without necessarily having to use sharing. So + 1 for keeping
> them in one service but exposing group only API’s to go along with User
> Profile API’s.
> >
> > + 1 for OAuth2 Client Credentials as an authentication mechanism
> between Gateways and Custos. Since Gateways are hosted on a secured
> services, this might work well.
> >
> > + 1 for Airavata proxy API’s for Custos functionality. This might
> help preserve usability of Airavata API’s while allowing independent use of
> Custos Services.
> >
> > Suresh
> >
> >> On Aug 19, 2019, at 5:09 PM, Christie, Marcus Aaron <
> machrist@iu.edu> wrote:
> >>
> >> Thanks for the update Aarushi. Replies inline:
> >>
> >> On 8/18/19, 6:25 PM, "Bisht, Aarushi" <ab...@iu.edu> wrote:
> >>
> >> 1. ??Group Service Migration:??? This service is tightly
> coupled with the airavata sharing registry service. Since sharing service
> will also be migrated to custos, I could defer the migration of group
> service until the migration of sharing service while keeping the dependency
> between these 2, or, I could decouple this dependency and make group
> service an independent service. We might need independent group service if
> we are allowing other Gateways to use only Custos authentication and
> profile service while implementing their own sharing service. What are our
> plans regarding this?
> >>
> >>
> >> The GroupManagerService essentially just adds authentication on the
> group functionality in the sharing service. I think they are inherently
> coupled. I'm assuming Custos will need some similar public API to its
> functionality.
> >>
> >>
> >> 2. Integration of Custos with Airavata: There are couple of
> discussions that I wanted to bring up:
> >>
> >> * ? Authentication between Gateways and Custos : As of now, I
> have not implemented any authentication/authorization between these 2
> services. Do we need this or would we rely on network and firewall based
> limited access to custos servers? If we do need it, I would appreciate
> suggestions on type of authentication to use.
> >>
> >>
> >> Since Custos will be potentially consumed by multiple non-Airavata
> gateways, I think it needs to authenticate/authorize calls. For
> service-to-service type calls my initial thought is to use OAuth2 client
> credentials.
> >>
> >>
> >> * ??Airavata proxying Custos services: Since the portals
> using Airavata should not be impacted with the migration of some of
> Airavata services, the Airavata APIs will essentially be proxying Custos
> services in some cases. For example the Airavata APIs for user
> creation,deletion, updation will remain the same but most of the business
> logic and database will be in Custos. Airavata will only keep user/tenant
> information specific to itself. So essentially, Airavata is just proxying
> Custos. This is true for most of the APIs which rely on Custos.
> >>
> >>
> >> I think proxying from the Airavata services to Custos sounds like a
> good idea.
> >
>
>
>
--
Best Regards,
Thejaka Amila Kanewala, PhD
https://github.com/thejkane/agm