You are viewing a plain text version of this content. The canonical link for it is here.
Posted to custos@airavata.apache.org by DImuthu Upeksha <di...@gmail.com> on 2019/07/08 16:52:29 UTC
Re: Module structure
Hi Amila, Aarushi, All,
Thanks for the clarification. I'm working on revamping current credential
store to support multiple credential types with the Vault [1] as the
backend. I will start a new thread to discuss more on that with a PR.
However do we have an idea on how our external APIs should be? Would that
be Thrift based or do we have any other reasons to look for an alternative
approach. I personally +0 for Thrift and +1 for a REST based approach
because users can easily understand the usage by looking at the
documentation. What do you think?
[1] https://www.vaultproject.io/
Thanks
Dimuthu
On Wed, Jun 26, 2019 at 3:27 AM Thejaka Amila J Kanewala <
thejaka.amila@gmail.com> wrote:
> Hi Aarushi,
>
> Well thought questions.
> Some answers/comments inline.
>
> Thanks
> -Thejaka
>
> On Tue, Jun 25, 2019 at 6:38 AM Bisht, Aarushi <ab...@iu.edu> wrote:
>
> > Hi Thejaka,
> >
> > Thank you for the project structure. Besides the above-mentioned modules,
> > I have also added
> >
> > 1. custos-commons - for common utility methods used across custos
> services
> > 2. custos-profile-service - which has profile service APIs, handlers and
> > other utility function for user profile service, tenant profile service
> and
> > iamadminservice
> > 3. thrift-interfaces - thrift files for all models and custos APIs
> >
> > Please let me know if any reorganization is required for the above
> modules.
> >
>
> This is good. We need these.
>
>
> >
> > I also have a couple of doubts related to some of the points mentioned in
> > the mail.
> >
> > 1. As you mentioned that Custos will provide multi-tenancy
> > functionalities. If I understand correctly, this means multiple tenants
> can
> > be initialized in Custos using Tenant Profile Service(which is currently
> > present in Airavata). Since I am migrating this service to custos, I want
> > to understand what does it mean for Airavata. Since Airavata will use
> > custos profile services, does this mean all tenant operation to Airavata
> > will be delegated to Custos?
> >
>
> Yes (Assuming "tenant operation" is referring to add/delete/modify tenants,
> add users to tenants, etc.). Airavata should only work with a reference to
> the tenant (e.g., tenant id). However, we need to make sure tenant
> isolation is properly implemented in Airavata (i.e., users in tenant1
> should not be able to access any resource belong to a user in tenant2
> (unless if they are shared)).
>
>
> >
> > Also, I am a little confused with the terminology. Airavata is a gateway,
> > what does allowing multi-tenants in airavata mean? When "public Gateway
> > addGateway(AuthzToken authzToken, Gateway gateway)" of
> TenantProfileService
> > is invoked, does gateway here refers to tenants like PGA, Seagrid?
> >
>
> This is a very good question and I believe Marlon already answered this. As
> Marlon described, Airavata is a gateway middleware. Scientists can use
> Airavata to build gateways. Each tenant is a separate gateway.
>
> "does gateway here refer to tenants like PGA, Seagrid?" -- correct.
>
>
> > 2. There are modules related to authentication and authorization in
> > Custos. I can understand the new authenticate flow but I am not very
> clear
> > about authorization. When a user invokes an API call, its authentication
> > will be delegated by Airavata to Custos which will authenticate the user
> > using the authztoken with Keycloak and return the result to Airavata.
> >
> > In case of authorization, every Airavata API call is authorized based on
> > the group user belongs to, which is either AdminGroup,
> ReadsOnlyAdminGroup
> > or a gateway user. The user group information and what groups are
> > authorized for which API call is kept with Airavata. I do not understand
> > how will authorization in Custos work?
> >
>
> Good question and I am not 100% sure whether I know the answer for this. I
> have not touched Airavata code base for a while and I don't clearly
> understand why we maintain groups within Airavata (why not in Keycloak
> groups and roles?). Maybe Suresh/Dimuthu or Marlon can explain why we
> maintain groups in Airavata but not in Keycloak.
>
> The way I envision the authorization functionality should be implemented is
> as follows:
>
> The authorization is performed on a particular resource. A resource can be
> represented using a unique string (sometimes called "permission string")
> and this string can be associated with an API function. Sometimes these
> resources hierarchical (e.g., a directory structure) in such cases those
> string can also be hierarchical ("api/addUser" etc.). Then, there is an
> action that can be performed on a resource (e.g., "read/write/invoke"
> etc.). In an API call, we need to make sure the current user has "invoke"
> permission on the resource associated with the API. So, in summary, Custos
> need to provide an API like following for authorization:
>
> "bool isAuthorize(tenantId, userId, permissionString, action)"
>
> Airavata should invoke above Custos API upon invoking the Airavata API. The
> underlying authorization method is irrelevant to Airavata (i.e., whether
> Custos is using RBAC or fine-grained authorization etc.) and possibly above
> method should inspect all inheriting permissions through groups, roles and
> sharing as well.
>
>
> > Also, I am currently following the first approach to integrate Keycloak
> > with Custos ie using keycloak as an external service as Airavata was
> using
> > this approach. Let me know if this needs to be changed.
> >
>
> +1, let's use this approach.
>
>
> >
> > Thanks & Regards,
> > Aarushi Bisht
> > ________________________________________
> > From: Thejaka Amila J Kanewala <th...@gmail.com>
> > Sent: Monday, June 24, 2019 1:07 PM
> > To: custos@airavata.apache.org
> > Subject: Re: Module structure
> >
> > Hi Dimuthu,
> >
> > I am sorry for the late reply. I thought I play around this first and
> reply
> > to your email but I got busy during the weekend and did not get a chance
> to
> > experiment around this. Therefore, I do not want to hold you on this
> > anymore.
> >
> > Here is what I had in mind:
> >
> > Custos provides the following functionalities:
> >
> > 1. Delegate science gateway authentication
> > 2. Provide transparent resource authentication to science gateways
> > 3. Provide sharing/multi-tenancy functionalities (what is already in
> > Airavata)
> >
> > To provide some of these functionalities we need an identity server like
> > Keycloak. So there are two ways we can integrate Keycloak with Custos: 1.
> > As an external service, 2. As an embedded server running inside Custos
> > server.
> > We had different opinions about these two approaches, however, I favored
> > the second approach as it simplifies the deployment complexity (and I
> guess
> > Suresh favored the first approach). The issue with the second approach is
> > there is no straightforward way to run Keycloak inside Tomcat as an
> > embedded server. There are a couple of blog posts saying that it is
> > possible however, I did not get enough time to successfully run them
> > (theoretically this is possible).
> >
> > So, back to your question about the structure. Assuming we take the first
> > approach (deploy Keycloak as an external server), I would imagine the
> > directory structure would be something similar to the following:
> >
> > modules
> > - custos-connectors (Connectors to connect to Keycloak or any other
> > identity server)
> > - custos-authentication (Delegated authentication)
> > - custos-authorisation
> > - custos-resource-authentication (Resource authentication e.g., Amazon,
> > Google clouds, SSH etc)
> > - custos-sharing (Airavata sharing)
> > - custos-credential-store
> > - custos-dist (Distribution + tomcat server)
> > - custos-client (Custos client library which will be used by science
> > gateways)
> > - custos-test (Integration tests)
> >
> > If we use the second approach, in addition to above modules, we will
> have a
> > module called "custos-server" which includes changes to embed keycloak
> > within Tomcat.
> >
> > In either case, the above-described structure is not static, it's just a
> > starting point. We definitely need to re-organize as we develop
> > functionality.
> >
> > Thanks
> > Best Regards,
> > Thejaka Amila Kanewala, PhD
> > https://github.com/thejkane/agm
> >
> > On Sun, Jun 16, 2019 at 9:17 PM DImuthu Upeksha <
> > dimuthu.upeksha2@gmail.com>
> > wrote:
> >
> > > Hi Folks,
> > >
> > > Is there a decided module structure for the project? If you have any
> > > suggestion, please let me know. I'm working on refactoring credential
> > store
> > > to support multiple protocols (SSH, S3, Azure, Box and etc) and I
> believe
> > > that this might me useful for Custos as well.
> > >
> > > Thanks
> > > Dimuthu
> > >
> >
> >
> > --
> >
>
>
> --
> Best Regards,
> Thejaka Amila Kanewala, PhD
> https://github.com/thejkane/agm
>
Re: Module structure
Posted by DImuthu Upeksha <di...@gmail.com>.
Hi All,
I have submitted a PR [2] with initial framework for credential store and
its API. Credential backend it using a Hashicorp Vault [3] store and API is
implemented as a REST API. Application is implemented as a Spring Boot
application with dependency injection for much better flexibility among
components.
[2] https://github.com/apache/airavata-custos/pull/3
[3] https://www.vaultproject.io/
Thanks
Dimuthu
On Tue, Jul 9, 2019 at 3:37 PM Thejaka Amila J Kanewala <
thejaka.amila@gmail.com> wrote:
> I am indifferent about the choice. Honestly, i dont have much
> recent experience with either, though i see lot of people use REST. So
> chose whichever you are comfortable with.
>
> Thejaka
>
> On Mon, Jul 8, 2019, 8:11 PM Suresh Marru <sm...@apache.org> wrote:
>
> > Hi Dimuthu,
> >
> > I am + 0 if you choose to go with REST.
> >
> > For Airavata the data models are complex and gateways use cases were very
> > diverse which made thrift a good choice. For Custos in general and the
> > credential store API in particular, I agree REST might be a good choice.
> >
> > Suresh
> >
> > > On Jul 8, 2019, at 12:52 PM, DImuthu Upeksha <
> dimuthu.upeksha2@gmail.com>
> > wrote:
> > >
> > > Hi Amila, Aarushi, All,
> > >
> > > Thanks for the clarification. I'm working on revamping current
> credential
> > > store to support multiple credential types with the Vault [1] as the
> > > backend. I will start a new thread to discuss more on that with a PR.
> > > However do we have an idea on how our external APIs should be? Would
> that
> > > be Thrift based or do we have any other reasons to look for an
> > alternative
> > > approach. I personally +0 for Thrift and +1 for a REST based approach
> > > because users can easily understand the usage by looking at the
> > > documentation. What do you think?
> > >
> > > [1] https://www.vaultproject.io/
> > >
> > > Thanks
> > > Dimuthu
> > >
> > > On Wed, Jun 26, 2019 at 3:27 AM Thejaka Amila J Kanewala <
> > > thejaka.amila@gmail.com> wrote:
> > >
> > >> Hi Aarushi,
> > >>
> > >> Well thought questions.
> > >> Some answers/comments inline.
> > >>
> > >> Thanks
> > >> -Thejaka
> > >>
> > >> On Tue, Jun 25, 2019 at 6:38 AM Bisht, Aarushi <ab...@iu.edu> wrote:
> > >>
> > >>> Hi Thejaka,
> > >>>
> > >>> Thank you for the project structure. Besides the above-mentioned
> > modules,
> > >>> I have also added
> > >>>
> > >>> 1. custos-commons - for common utility methods used across custos
> > >> services
> > >>> 2. custos-profile-service - which has profile service APIs, handlers
> > and
> > >>> other utility function for user profile service, tenant profile
> service
> > >> and
> > >>> iamadminservice
> > >>> 3. thrift-interfaces - thrift files for all models and custos APIs
> > >>>
> > >>> Please let me know if any reorganization is required for the above
> > >> modules.
> > >>>
> > >>
> > >> This is good. We need these.
> > >>
> > >>
> > >>>
> > >>> I also have a couple of doubts related to some of the points
> mentioned
> > in
> > >>> the mail.
> > >>>
> > >>> 1. As you mentioned that Custos will provide multi-tenancy
> > >>> functionalities. If I understand correctly, this means multiple
> tenants
> > >> can
> > >>> be initialized in Custos using Tenant Profile Service(which is
> > currently
> > >>> present in Airavata). Since I am migrating this service to custos, I
> > want
> > >>> to understand what does it mean for Airavata. Since Airavata will use
> > >>> custos profile services, does this mean all tenant operation to
> > Airavata
> > >>> will be delegated to Custos?
> > >>>
> > >>
> > >> Yes (Assuming "tenant operation" is referring to add/delete/modify
> > tenants,
> > >> add users to tenants, etc.). Airavata should only work with a
> reference
> > to
> > >> the tenant (e.g., tenant id). However, we need to make sure tenant
> > >> isolation is properly implemented in Airavata (i.e., users in tenant1
> > >> should not be able to access any resource belong to a user in tenant2
> > >> (unless if they are shared)).
> > >>
> > >>
> > >>>
> > >>> Also, I am a little confused with the terminology. Airavata is a
> > gateway,
> > >>> what does allowing multi-tenants in airavata mean? When "public
> Gateway
> > >>> addGateway(AuthzToken authzToken, Gateway gateway)" of
> > >> TenantProfileService
> > >>> is invoked, does gateway here refers to tenants like PGA, Seagrid?
> > >>>
> > >>
> > >> This is a very good question and I believe Marlon already answered
> > this. As
> > >> Marlon described, Airavata is a gateway middleware. Scientists can use
> > >> Airavata to build gateways. Each tenant is a separate gateway.
> > >>
> > >> "does gateway here refer to tenants like PGA, Seagrid?" -- correct.
> > >>
> > >>
> > >>> 2. There are modules related to authentication and authorization in
> > >>> Custos. I can understand the new authenticate flow but I am not very
> > >> clear
> > >>> about authorization. When a user invokes an API call, its
> > authentication
> > >>> will be delegated by Airavata to Custos which will authenticate the
> > user
> > >>> using the authztoken with Keycloak and return the result to Airavata.
> > >>>
> > >>> In case of authorization, every Airavata API call is authorized based
> > on
> > >>> the group user belongs to, which is either AdminGroup,
> > >> ReadsOnlyAdminGroup
> > >>> or a gateway user. The user group information and what groups are
> > >>> authorized for which API call is kept with Airavata. I do not
> > understand
> > >>> how will authorization in Custos work?
> > >>>
> > >>
> > >> Good question and I am not 100% sure whether I know the answer for
> > this. I
> > >> have not touched Airavata code base for a while and I don't clearly
> > >> understand why we maintain groups within Airavata (why not in Keycloak
> > >> groups and roles?). Maybe Suresh/Dimuthu or Marlon can explain why we
> > >> maintain groups in Airavata but not in Keycloak.
> > >>
> > >> The way I envision the authorization functionality should be
> > implemented is
> > >> as follows:
> > >>
> > >> The authorization is performed on a particular resource. A resource
> can
> > be
> > >> represented using a unique string (sometimes called "permission
> string")
> > >> and this string can be associated with an API function. Sometimes
> these
> > >> resources hierarchical (e.g., a directory structure) in such cases
> those
> > >> string can also be hierarchical ("api/addUser" etc.). Then, there is
> an
> > >> action that can be performed on a resource (e.g., "read/write/invoke"
> > >> etc.). In an API call, we need to make sure the current user has
> > "invoke"
> > >> permission on the resource associated with the API. So, in summary,
> > Custos
> > >> need to provide an API like following for authorization:
> > >>
> > >> "bool isAuthorize(tenantId, userId, permissionString, action)"
> > >>
> > >> Airavata should invoke above Custos API upon invoking the Airavata
> API.
> > The
> > >> underlying authorization method is irrelevant to Airavata (i.e.,
> whether
> > >> Custos is using RBAC or fine-grained authorization etc.) and possibly
> > above
> > >> method should inspect all inheriting permissions through groups, roles
> > and
> > >> sharing as well.
> > >>
> > >>
> > >>> Also, I am currently following the first approach to integrate
> Keycloak
> > >>> with Custos ie using keycloak as an external service as Airavata was
> > >> using
> > >>> this approach. Let me know if this needs to be changed.
> > >>>
> > >>
> > >> +1, let's use this approach.
> > >>
> > >>
> > >>>
> > >>> Thanks & Regards,
> > >>> Aarushi Bisht
> > >>> ________________________________________
> > >>> From: Thejaka Amila J Kanewala <th...@gmail.com>
> > >>> Sent: Monday, June 24, 2019 1:07 PM
> > >>> To: custos@airavata.apache.org
> > >>> Subject: Re: Module structure
> > >>>
> > >>> Hi Dimuthu,
> > >>>
> > >>> I am sorry for the late reply. I thought I play around this first and
> > >> reply
> > >>> to your email but I got busy during the weekend and did not get a
> > chance
> > >> to
> > >>> experiment around this. Therefore, I do not want to hold you on this
> > >>> anymore.
> > >>>
> > >>> Here is what I had in mind:
> > >>>
> > >>> Custos provides the following functionalities:
> > >>>
> > >>> 1. Delegate science gateway authentication
> > >>> 2. Provide transparent resource authentication to science gateways
> > >>> 3. Provide sharing/multi-tenancy functionalities (what is already in
> > >>> Airavata)
> > >>>
> > >>> To provide some of these functionalities we need an identity server
> > like
> > >>> Keycloak. So there are two ways we can integrate Keycloak with
> Custos:
> > 1.
> > >>> As an external service, 2. As an embedded server running inside
> Custos
> > >>> server.
> > >>> We had different opinions about these two approaches, however, I
> > favored
> > >>> the second approach as it simplifies the deployment complexity (and I
> > >> guess
> > >>> Suresh favored the first approach). The issue with the second
> approach
> > is
> > >>> there is no straightforward way to run Keycloak inside Tomcat as an
> > >>> embedded server. There are a couple of blog posts saying that it is
> > >>> possible however, I did not get enough time to successfully run them
> > >>> (theoretically this is possible).
> > >>>
> > >>> So, back to your question about the structure. Assuming we take the
> > first
> > >>> approach (deploy Keycloak as an external server), I would imagine the
> > >>> directory structure would be something similar to the following:
> > >>>
> > >>> modules
> > >>> - custos-connectors (Connectors to connect to Keycloak or any other
> > >>> identity server)
> > >>> - custos-authentication (Delegated authentication)
> > >>> - custos-authorisation
> > >>> - custos-resource-authentication (Resource authentication e.g.,
> Amazon,
> > >>> Google clouds, SSH etc)
> > >>> - custos-sharing (Airavata sharing)
> > >>> - custos-credential-store
> > >>> - custos-dist (Distribution + tomcat server)
> > >>> - custos-client (Custos client library which will be used by science
> > >>> gateways)
> > >>> - custos-test (Integration tests)
> > >>>
> > >>> If we use the second approach, in addition to above modules, we will
> > >> have a
> > >>> module called "custos-server" which includes changes to embed
> keycloak
> > >>> within Tomcat.
> > >>>
> > >>> In either case, the above-described structure is not static, it's
> just
> > a
> > >>> starting point. We definitely need to re-organize as we develop
> > >>> functionality.
> > >>>
> > >>> Thanks
> > >>> Best Regards,
> > >>> Thejaka Amila Kanewala, PhD
> > >>> https://github.com/thejkane/agm
> > >>>
> > >>> On Sun, Jun 16, 2019 at 9:17 PM DImuthu Upeksha <
> > >>> dimuthu.upeksha2@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> Hi Folks,
> > >>>>
> > >>>> Is there a decided module structure for the project? If you have any
> > >>>> suggestion, please let me know. I'm working on refactoring
> credential
> > >>> store
> > >>>> to support multiple protocols (SSH, S3, Azure, Box and etc) and I
> > >> believe
> > >>>> that this might me useful for Custos as well.
> > >>>>
> > >>>> Thanks
> > >>>> Dimuthu
> > >>>>
> > >>>
> > >>>
> > >>> --
> > >>>
> > >>
> > >>
> > >> --
> > >> Best Regards,
> > >> Thejaka Amila Kanewala, PhD
> > >> https://github.com/thejkane/agm
> > >>
> >
> >
>
Re: Module structure
Posted by Thejaka Amila J Kanewala <th...@gmail.com>.
I am indifferent about the choice. Honestly, i dont have much
recent experience with either, though i see lot of people use REST. So
chose whichever you are comfortable with.
Thejaka
On Mon, Jul 8, 2019, 8:11 PM Suresh Marru <sm...@apache.org> wrote:
> Hi Dimuthu,
>
> I am + 0 if you choose to go with REST.
>
> For Airavata the data models are complex and gateways use cases were very
> diverse which made thrift a good choice. For Custos in general and the
> credential store API in particular, I agree REST might be a good choice.
>
> Suresh
>
> > On Jul 8, 2019, at 12:52 PM, DImuthu Upeksha <di...@gmail.com>
> wrote:
> >
> > Hi Amila, Aarushi, All,
> >
> > Thanks for the clarification. I'm working on revamping current credential
> > store to support multiple credential types with the Vault [1] as the
> > backend. I will start a new thread to discuss more on that with a PR.
> > However do we have an idea on how our external APIs should be? Would that
> > be Thrift based or do we have any other reasons to look for an
> alternative
> > approach. I personally +0 for Thrift and +1 for a REST based approach
> > because users can easily understand the usage by looking at the
> > documentation. What do you think?
> >
> > [1] https://www.vaultproject.io/
> >
> > Thanks
> > Dimuthu
> >
> > On Wed, Jun 26, 2019 at 3:27 AM Thejaka Amila J Kanewala <
> > thejaka.amila@gmail.com> wrote:
> >
> >> Hi Aarushi,
> >>
> >> Well thought questions.
> >> Some answers/comments inline.
> >>
> >> Thanks
> >> -Thejaka
> >>
> >> On Tue, Jun 25, 2019 at 6:38 AM Bisht, Aarushi <ab...@iu.edu> wrote:
> >>
> >>> Hi Thejaka,
> >>>
> >>> Thank you for the project structure. Besides the above-mentioned
> modules,
> >>> I have also added
> >>>
> >>> 1. custos-commons - for common utility methods used across custos
> >> services
> >>> 2. custos-profile-service - which has profile service APIs, handlers
> and
> >>> other utility function for user profile service, tenant profile service
> >> and
> >>> iamadminservice
> >>> 3. thrift-interfaces - thrift files for all models and custos APIs
> >>>
> >>> Please let me know if any reorganization is required for the above
> >> modules.
> >>>
> >>
> >> This is good. We need these.
> >>
> >>
> >>>
> >>> I also have a couple of doubts related to some of the points mentioned
> in
> >>> the mail.
> >>>
> >>> 1. As you mentioned that Custos will provide multi-tenancy
> >>> functionalities. If I understand correctly, this means multiple tenants
> >> can
> >>> be initialized in Custos using Tenant Profile Service(which is
> currently
> >>> present in Airavata). Since I am migrating this service to custos, I
> want
> >>> to understand what does it mean for Airavata. Since Airavata will use
> >>> custos profile services, does this mean all tenant operation to
> Airavata
> >>> will be delegated to Custos?
> >>>
> >>
> >> Yes (Assuming "tenant operation" is referring to add/delete/modify
> tenants,
> >> add users to tenants, etc.). Airavata should only work with a reference
> to
> >> the tenant (e.g., tenant id). However, we need to make sure tenant
> >> isolation is properly implemented in Airavata (i.e., users in tenant1
> >> should not be able to access any resource belong to a user in tenant2
> >> (unless if they are shared)).
> >>
> >>
> >>>
> >>> Also, I am a little confused with the terminology. Airavata is a
> gateway,
> >>> what does allowing multi-tenants in airavata mean? When "public Gateway
> >>> addGateway(AuthzToken authzToken, Gateway gateway)" of
> >> TenantProfileService
> >>> is invoked, does gateway here refers to tenants like PGA, Seagrid?
> >>>
> >>
> >> This is a very good question and I believe Marlon already answered
> this. As
> >> Marlon described, Airavata is a gateway middleware. Scientists can use
> >> Airavata to build gateways. Each tenant is a separate gateway.
> >>
> >> "does gateway here refer to tenants like PGA, Seagrid?" -- correct.
> >>
> >>
> >>> 2. There are modules related to authentication and authorization in
> >>> Custos. I can understand the new authenticate flow but I am not very
> >> clear
> >>> about authorization. When a user invokes an API call, its
> authentication
> >>> will be delegated by Airavata to Custos which will authenticate the
> user
> >>> using the authztoken with Keycloak and return the result to Airavata.
> >>>
> >>> In case of authorization, every Airavata API call is authorized based
> on
> >>> the group user belongs to, which is either AdminGroup,
> >> ReadsOnlyAdminGroup
> >>> or a gateway user. The user group information and what groups are
> >>> authorized for which API call is kept with Airavata. I do not
> understand
> >>> how will authorization in Custos work?
> >>>
> >>
> >> Good question and I am not 100% sure whether I know the answer for
> this. I
> >> have not touched Airavata code base for a while and I don't clearly
> >> understand why we maintain groups within Airavata (why not in Keycloak
> >> groups and roles?). Maybe Suresh/Dimuthu or Marlon can explain why we
> >> maintain groups in Airavata but not in Keycloak.
> >>
> >> The way I envision the authorization functionality should be
> implemented is
> >> as follows:
> >>
> >> The authorization is performed on a particular resource. A resource can
> be
> >> represented using a unique string (sometimes called "permission string")
> >> and this string can be associated with an API function. Sometimes these
> >> resources hierarchical (e.g., a directory structure) in such cases those
> >> string can also be hierarchical ("api/addUser" etc.). Then, there is an
> >> action that can be performed on a resource (e.g., "read/write/invoke"
> >> etc.). In an API call, we need to make sure the current user has
> "invoke"
> >> permission on the resource associated with the API. So, in summary,
> Custos
> >> need to provide an API like following for authorization:
> >>
> >> "bool isAuthorize(tenantId, userId, permissionString, action)"
> >>
> >> Airavata should invoke above Custos API upon invoking the Airavata API.
> The
> >> underlying authorization method is irrelevant to Airavata (i.e., whether
> >> Custos is using RBAC or fine-grained authorization etc.) and possibly
> above
> >> method should inspect all inheriting permissions through groups, roles
> and
> >> sharing as well.
> >>
> >>
> >>> Also, I am currently following the first approach to integrate Keycloak
> >>> with Custos ie using keycloak as an external service as Airavata was
> >> using
> >>> this approach. Let me know if this needs to be changed.
> >>>
> >>
> >> +1, let's use this approach.
> >>
> >>
> >>>
> >>> Thanks & Regards,
> >>> Aarushi Bisht
> >>> ________________________________________
> >>> From: Thejaka Amila J Kanewala <th...@gmail.com>
> >>> Sent: Monday, June 24, 2019 1:07 PM
> >>> To: custos@airavata.apache.org
> >>> Subject: Re: Module structure
> >>>
> >>> Hi Dimuthu,
> >>>
> >>> I am sorry for the late reply. I thought I play around this first and
> >> reply
> >>> to your email but I got busy during the weekend and did not get a
> chance
> >> to
> >>> experiment around this. Therefore, I do not want to hold you on this
> >>> anymore.
> >>>
> >>> Here is what I had in mind:
> >>>
> >>> Custos provides the following functionalities:
> >>>
> >>> 1. Delegate science gateway authentication
> >>> 2. Provide transparent resource authentication to science gateways
> >>> 3. Provide sharing/multi-tenancy functionalities (what is already in
> >>> Airavata)
> >>>
> >>> To provide some of these functionalities we need an identity server
> like
> >>> Keycloak. So there are two ways we can integrate Keycloak with Custos:
> 1.
> >>> As an external service, 2. As an embedded server running inside Custos
> >>> server.
> >>> We had different opinions about these two approaches, however, I
> favored
> >>> the second approach as it simplifies the deployment complexity (and I
> >> guess
> >>> Suresh favored the first approach). The issue with the second approach
> is
> >>> there is no straightforward way to run Keycloak inside Tomcat as an
> >>> embedded server. There are a couple of blog posts saying that it is
> >>> possible however, I did not get enough time to successfully run them
> >>> (theoretically this is possible).
> >>>
> >>> So, back to your question about the structure. Assuming we take the
> first
> >>> approach (deploy Keycloak as an external server), I would imagine the
> >>> directory structure would be something similar to the following:
> >>>
> >>> modules
> >>> - custos-connectors (Connectors to connect to Keycloak or any other
> >>> identity server)
> >>> - custos-authentication (Delegated authentication)
> >>> - custos-authorisation
> >>> - custos-resource-authentication (Resource authentication e.g., Amazon,
> >>> Google clouds, SSH etc)
> >>> - custos-sharing (Airavata sharing)
> >>> - custos-credential-store
> >>> - custos-dist (Distribution + tomcat server)
> >>> - custos-client (Custos client library which will be used by science
> >>> gateways)
> >>> - custos-test (Integration tests)
> >>>
> >>> If we use the second approach, in addition to above modules, we will
> >> have a
> >>> module called "custos-server" which includes changes to embed keycloak
> >>> within Tomcat.
> >>>
> >>> In either case, the above-described structure is not static, it's just
> a
> >>> starting point. We definitely need to re-organize as we develop
> >>> functionality.
> >>>
> >>> Thanks
> >>> Best Regards,
> >>> Thejaka Amila Kanewala, PhD
> >>> https://github.com/thejkane/agm
> >>>
> >>> On Sun, Jun 16, 2019 at 9:17 PM DImuthu Upeksha <
> >>> dimuthu.upeksha2@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi Folks,
> >>>>
> >>>> Is there a decided module structure for the project? If you have any
> >>>> suggestion, please let me know. I'm working on refactoring credential
> >>> store
> >>>> to support multiple protocols (SSH, S3, Azure, Box and etc) and I
> >> believe
> >>>> that this might me useful for Custos as well.
> >>>>
> >>>> Thanks
> >>>> Dimuthu
> >>>>
> >>>
> >>>
> >>> --
> >>>
> >>
> >>
> >> --
> >> Best Regards,
> >> Thejaka Amila Kanewala, PhD
> >> https://github.com/thejkane/agm
> >>
>
>
Re: Module structure
Posted by Suresh Marru <sm...@apache.org>.
Hi Dimuthu,
I am + 0 if you choose to go with REST.
For Airavata the data models are complex and gateways use cases were very diverse which made thrift a good choice. For Custos in general and the credential store API in particular, I agree REST might be a good choice.
Suresh
> On Jul 8, 2019, at 12:52 PM, DImuthu Upeksha <di...@gmail.com> wrote:
>
> Hi Amila, Aarushi, All,
>
> Thanks for the clarification. I'm working on revamping current credential
> store to support multiple credential types with the Vault [1] as the
> backend. I will start a new thread to discuss more on that with a PR.
> However do we have an idea on how our external APIs should be? Would that
> be Thrift based or do we have any other reasons to look for an alternative
> approach. I personally +0 for Thrift and +1 for a REST based approach
> because users can easily understand the usage by looking at the
> documentation. What do you think?
>
> [1] https://www.vaultproject.io/
>
> Thanks
> Dimuthu
>
> On Wed, Jun 26, 2019 at 3:27 AM Thejaka Amila J Kanewala <
> thejaka.amila@gmail.com> wrote:
>
>> Hi Aarushi,
>>
>> Well thought questions.
>> Some answers/comments inline.
>>
>> Thanks
>> -Thejaka
>>
>> On Tue, Jun 25, 2019 at 6:38 AM Bisht, Aarushi <ab...@iu.edu> wrote:
>>
>>> Hi Thejaka,
>>>
>>> Thank you for the project structure. Besides the above-mentioned modules,
>>> I have also added
>>>
>>> 1. custos-commons - for common utility methods used across custos
>> services
>>> 2. custos-profile-service - which has profile service APIs, handlers and
>>> other utility function for user profile service, tenant profile service
>> and
>>> iamadminservice
>>> 3. thrift-interfaces - thrift files for all models and custos APIs
>>>
>>> Please let me know if any reorganization is required for the above
>> modules.
>>>
>>
>> This is good. We need these.
>>
>>
>>>
>>> I also have a couple of doubts related to some of the points mentioned in
>>> the mail.
>>>
>>> 1. As you mentioned that Custos will provide multi-tenancy
>>> functionalities. If I understand correctly, this means multiple tenants
>> can
>>> be initialized in Custos using Tenant Profile Service(which is currently
>>> present in Airavata). Since I am migrating this service to custos, I want
>>> to understand what does it mean for Airavata. Since Airavata will use
>>> custos profile services, does this mean all tenant operation to Airavata
>>> will be delegated to Custos?
>>>
>>
>> Yes (Assuming "tenant operation" is referring to add/delete/modify tenants,
>> add users to tenants, etc.). Airavata should only work with a reference to
>> the tenant (e.g., tenant id). However, we need to make sure tenant
>> isolation is properly implemented in Airavata (i.e., users in tenant1
>> should not be able to access any resource belong to a user in tenant2
>> (unless if they are shared)).
>>
>>
>>>
>>> Also, I am a little confused with the terminology. Airavata is a gateway,
>>> what does allowing multi-tenants in airavata mean? When "public Gateway
>>> addGateway(AuthzToken authzToken, Gateway gateway)" of
>> TenantProfileService
>>> is invoked, does gateway here refers to tenants like PGA, Seagrid?
>>>
>>
>> This is a very good question and I believe Marlon already answered this. As
>> Marlon described, Airavata is a gateway middleware. Scientists can use
>> Airavata to build gateways. Each tenant is a separate gateway.
>>
>> "does gateway here refer to tenants like PGA, Seagrid?" -- correct.
>>
>>
>>> 2. There are modules related to authentication and authorization in
>>> Custos. I can understand the new authenticate flow but I am not very
>> clear
>>> about authorization. When a user invokes an API call, its authentication
>>> will be delegated by Airavata to Custos which will authenticate the user
>>> using the authztoken with Keycloak and return the result to Airavata.
>>>
>>> In case of authorization, every Airavata API call is authorized based on
>>> the group user belongs to, which is either AdminGroup,
>> ReadsOnlyAdminGroup
>>> or a gateway user. The user group information and what groups are
>>> authorized for which API call is kept with Airavata. I do not understand
>>> how will authorization in Custos work?
>>>
>>
>> Good question and I am not 100% sure whether I know the answer for this. I
>> have not touched Airavata code base for a while and I don't clearly
>> understand why we maintain groups within Airavata (why not in Keycloak
>> groups and roles?). Maybe Suresh/Dimuthu or Marlon can explain why we
>> maintain groups in Airavata but not in Keycloak.
>>
>> The way I envision the authorization functionality should be implemented is
>> as follows:
>>
>> The authorization is performed on a particular resource. A resource can be
>> represented using a unique string (sometimes called "permission string")
>> and this string can be associated with an API function. Sometimes these
>> resources hierarchical (e.g., a directory structure) in such cases those
>> string can also be hierarchical ("api/addUser" etc.). Then, there is an
>> action that can be performed on a resource (e.g., "read/write/invoke"
>> etc.). In an API call, we need to make sure the current user has "invoke"
>> permission on the resource associated with the API. So, in summary, Custos
>> need to provide an API like following for authorization:
>>
>> "bool isAuthorize(tenantId, userId, permissionString, action)"
>>
>> Airavata should invoke above Custos API upon invoking the Airavata API. The
>> underlying authorization method is irrelevant to Airavata (i.e., whether
>> Custos is using RBAC or fine-grained authorization etc.) and possibly above
>> method should inspect all inheriting permissions through groups, roles and
>> sharing as well.
>>
>>
>>> Also, I am currently following the first approach to integrate Keycloak
>>> with Custos ie using keycloak as an external service as Airavata was
>> using
>>> this approach. Let me know if this needs to be changed.
>>>
>>
>> +1, let's use this approach.
>>
>>
>>>
>>> Thanks & Regards,
>>> Aarushi Bisht
>>> ________________________________________
>>> From: Thejaka Amila J Kanewala <th...@gmail.com>
>>> Sent: Monday, June 24, 2019 1:07 PM
>>> To: custos@airavata.apache.org
>>> Subject: Re: Module structure
>>>
>>> Hi Dimuthu,
>>>
>>> I am sorry for the late reply. I thought I play around this first and
>> reply
>>> to your email but I got busy during the weekend and did not get a chance
>> to
>>> experiment around this. Therefore, I do not want to hold you on this
>>> anymore.
>>>
>>> Here is what I had in mind:
>>>
>>> Custos provides the following functionalities:
>>>
>>> 1. Delegate science gateway authentication
>>> 2. Provide transparent resource authentication to science gateways
>>> 3. Provide sharing/multi-tenancy functionalities (what is already in
>>> Airavata)
>>>
>>> To provide some of these functionalities we need an identity server like
>>> Keycloak. So there are two ways we can integrate Keycloak with Custos: 1.
>>> As an external service, 2. As an embedded server running inside Custos
>>> server.
>>> We had different opinions about these two approaches, however, I favored
>>> the second approach as it simplifies the deployment complexity (and I
>> guess
>>> Suresh favored the first approach). The issue with the second approach is
>>> there is no straightforward way to run Keycloak inside Tomcat as an
>>> embedded server. There are a couple of blog posts saying that it is
>>> possible however, I did not get enough time to successfully run them
>>> (theoretically this is possible).
>>>
>>> So, back to your question about the structure. Assuming we take the first
>>> approach (deploy Keycloak as an external server), I would imagine the
>>> directory structure would be something similar to the following:
>>>
>>> modules
>>> - custos-connectors (Connectors to connect to Keycloak or any other
>>> identity server)
>>> - custos-authentication (Delegated authentication)
>>> - custos-authorisation
>>> - custos-resource-authentication (Resource authentication e.g., Amazon,
>>> Google clouds, SSH etc)
>>> - custos-sharing (Airavata sharing)
>>> - custos-credential-store
>>> - custos-dist (Distribution + tomcat server)
>>> - custos-client (Custos client library which will be used by science
>>> gateways)
>>> - custos-test (Integration tests)
>>>
>>> If we use the second approach, in addition to above modules, we will
>> have a
>>> module called "custos-server" which includes changes to embed keycloak
>>> within Tomcat.
>>>
>>> In either case, the above-described structure is not static, it's just a
>>> starting point. We definitely need to re-organize as we develop
>>> functionality.
>>>
>>> Thanks
>>> Best Regards,
>>> Thejaka Amila Kanewala, PhD
>>> https://github.com/thejkane/agm
>>>
>>> On Sun, Jun 16, 2019 at 9:17 PM DImuthu Upeksha <
>>> dimuthu.upeksha2@gmail.com>
>>> wrote:
>>>
>>>> Hi Folks,
>>>>
>>>> Is there a decided module structure for the project? If you have any
>>>> suggestion, please let me know. I'm working on refactoring credential
>>> store
>>>> to support multiple protocols (SSH, S3, Azure, Box and etc) and I
>> believe
>>>> that this might me useful for Custos as well.
>>>>
>>>> Thanks
>>>> Dimuthu
>>>>
>>>
>>>
>>> --
>>>
>>
>>
>> --
>> Best Regards,
>> Thejaka Amila Kanewala, PhD
>> https://github.com/thejkane/agm
>>