[jira] [Comment Edited] (HDDS-4944) Multi-Tenant Support in Ozone

["Marton Elek (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDDS-4944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17316175#comment-17316175 ] 

Marton Elek edited comment on HDDS-4944 at 4/7/21, 9:43 AM:
------------------------------------------------------------

Thank you very much the requirement document @Prashant.

I think it's AWESOME. 

It's very helpful, very well written, and it helps to continue the discussion about the best possible implementations.

As I wrote earlier, still I am not convinced why do we combine the multi-tenancy feature with the kerberos-independent secret management. As they can be implemented independent of each other I still recommend separating the two question unless we have strong reason to not do it.

But back to the requirement list: the list is very good, because we can talk about the requirements and what solution can be used to implement them.

My overall feeling is that volume abstraction already provides enough features for these requirements, except the group/role management where we can consider other options (like investing more in integration instead of starting a complex user/group management).  At the same time I have concerns with adding more abstraction level which makes both the maintenance and the user experience significant more complex.

To explain it in more details, I added my comments to the requirement lines. 

bq. 2.a Ability to isolate users into different sub-organizations.

I think this is possible even today with proper group management. 

bq. 2.b There should be a namespace isolation between users of different sub-organizations. E.g. a user “xyz” can exist in “support” sub-organization and another user with same name “xyz” could exist in engineering. They would also have different access credentials

I am not sure about this one. I think it's rather confusing as we use Hadoop UGI. We should have a different UGI name for different identities to have proper ACL handling and audit logging. Which means that the two users should be different (with multi-tenant abstraction it means that we need to add the tenancy to the username). I would rather say that the AWS S3 compatible access key id should be as randomized as in AWS (and independent of UIG username) and we don't need to care about this one. 
 
bq. 2.c. Ability to restrict visibility of a user to resources (e.g. buckets, policies, users) only within their sub-org by default.

This is possible with volumes, even today. The only missing point is users, but we don't have user list at all. Listing users of a specific organization is more like a group management (see below)

bq. 2.d. Ability to further restrict users RW access to the buckets that they own. Other users of the same sub-org can see buckets within their sub-org as present but they can not read or write them.

This seems to be possible with group level ACL if I understood well.

bq. 2.e. Ability to do resource consolidation for all the users belonging to one sub-organization e.g. space usage by the sub-organization.

Today we have volumes for exactly the same reasons (AFAIK). We have volume level quota. Adding a new abstraction which also has similar functionalities can make the administration (and code maintenance) more complex IMHO.

bq. 2.f. Ability to do quota enforcement for the users of sub-organization, Ability to do QoS enforcement within a sub-organization.

Is it possible with dedicating volumes to sub-organizations?

bq. 2.h.Ability for some users to assume an Admin role within their sub-organization.

For me, this is the most important part. We already have a volume notion for bucket-namespace isolation, but this is missing from our current offering. This requires a full user/group management interface.

Again: this can be a question which is independent of the tenancy. We can have a group management feature where we maintain the role of the users (admin/super-admin...) and together with ACL/bucket policies we can provide similar features.

The question here: do we need to add this complex feature (full group management) to Ozone?

To be honest, I am not sure. I would say that there are other options here:

 * slightly improve the current s3 credential handling to make the credentials Kerberos independent
 * AND instead of investing in code for group/user management on our side, invest on extensibility and integration with other services (Keycloak, Hashicorp Vault, AWS STS, Minio STS) which can provide similar features in a more flexible way (or eventually create an Ozone STS similar to Minio STS).
 
 I have the fear that starting to implement a feature-full group management makes our maintenance burden very high.
 
  1. Even today we have a very good concept about recon as a UI, but we didn't have any time to invent to it. Based on this experience I am not sure if we can provide decent user experience for managing users.
  2. Even today, we had multiple problems with our security implementation. Adding additional complexity, layers and group/user management requires very careful planning and some level of security review ("audit").
  
> The proposal is to add this same and well known abstraction/terminology of “multi-tenant” environment for Ozone and fill the gaps as appropriate.

This is where I respectfully disagree. While I agree that it can be called as "multi-tenant", I am not sure if the implementation should be the addition of a new abstraction layer on top of the volumes as volumes are already created to provide very similar features. Therefore, (IMHO) we should consider to re-use our existing abstraction (volumes) as a solution for the multi-tenancy.

Volumes are independent of the account-namespace insolations, which is definitely a missing point. But based on the requirements, what we are looking for, is more like a user/group management feature ("tell me the user lists what I am managing..."), it's not a volume level user management.

My suggestions:

 * Investigate alternatives which can provide almost the same functionalities with significant less maintenance burden (and less implementation work !!!)
   * Using volume for bucket-namespace isolation instead of a new multi-tenancy abstraction layer
   * Using external group management / credential handling vs. investing in our user management (some investment may be required on our side, for making it more kerberos free, but it's an independent question) 
 * Compare the possible solutions with pro/con list (complexity, features, maintenance cost, ...)
 * Separate multi-tenancy and kerberos-independent secret management questions

Thank you again the answers.




was (Author: elek):
Thank you very much the requirement document @Prashant.

I think it's AWESOME. 

It's very helpful, very well written, and it helps to continue the discussion about the best possible implementations.

As I wrote earlier, still I am not convinced why do we combine the multi-tenancy feature with the kerberos-independent secret management. As they can be implemented independent of each other I still recommend separating the two question unless we have strong reason to not do it.

But back to the requirement list: the list is very good, because we can talk about the requirements and what solution can be used to implement them.

My overall feeling is that volume abstraction already provides enough features for these requirements, except the group/role management where we can consider other options (like investing more in integration instead of starting a complex user/group management).  At the same time I have concerns with adding more abstraction level which makes both the maintenance and the user experience significant more complex.

To explain it in more details, I added my comments to the requirement lines. 

bq. 2.a Ability to isolate users into different sub-organizations.

I think this is possible even today with proper group management. 

> 2.b There should be a namespace isolation between users of different sub-organizations. E.g. a user “xyz” can exist in “support” sub-organization and another user with same name “xyz” could exist in engineering. They would also have different access credentials

I am not sure about this one. I think it's rather confusing as we use Hadoop UGI. We should have a different UGI name for different identities to have proper ACL handling and audit logging. Which means that the two users should be different (with multi-tenant abstraction it means that we need to add the tenancy to the username). I would rather say that the AWS S3 compatible access key id should be as randomized as in AWS (and independent of UIG username) and we don't need to care about this one. 
 
> 2.c. Ability to restrict visibility of a user to resources (e.g. buckets, policies, users) only within their sub-org by default.

This is possible with volumes, even today. The only missing point is users, but we don't have user list at all. Listing users of specific organization is more like a group management (see bellow)

> 2.d. Ability to further restrict users RW access to the buckets that they own. Other users of the same sub-org can see buckets within their sub-org as present but they can not read or write them.

This seems to be possible with group level ACL if I understood well.

> 2.e. Ability to do resource consolidation for all the users belonging to one sub-organization e.g. space usage by the sub-organization.

Today we have volumes for exactly the same reasons (AFAIK). We have volume level quota. Adding a new abstraction which also has similar functionalities can make the administration (and code maintenance) more complex IMHO.

> 2.f. Ability to do quota enforcement for the users of sub-organization, Ability to do QoS enforcement within a sub-organization.

Is it possible with dedicating volumes to sub-organizations?

> 2.h.Ability for some users to assume an Admin role within their sub-organization.

For me, this is the most important part. We already have a volume notion for bucket-namespace isolation, but this is missing from our current offering. This requires a full user/group management interface.

Again: this can be a question which is independent of the tenancy. We can have a group management feature where we maintain the role of the users (admin/super-admin...) and together with ACL/bucket policies we can provide similar features.

The question here: do we need to add this complex feature (full group management) to Ozone?

To be honest, I am not sure. I would say that there are other options here:

 * slightly improve the current s3 credential handling to make the credentials Kerberos independent
 * AND instead of investing in code for group/user management on our side, invest on extensibility and integration with other services (Keycloak, Hashicorp Vault, AWS STS, Minio STS) which can provide similar features in a more flexible way (or eventually create an Ozone STS similar to Minio STS).
 
 I have the fear that starting to implement a feature-full group management makes our maintenance burden very high.
 
  1. Even today we have a very good concept about recon as a UI, but we didn't have any time to invent to it. Based on this experiance I am not sure if we can provide decent user experience for managing users.
  2. Even today, we had multiple problems with our security implementation. Adding additional complexity, layers and group/user management requires very careful planning and some level of security review ("audit").
  
> The proposal is to add this same and well known abstraction/terminology of “multi-tenant” environment for Ozone and fill the gaps as appropriate.

This is where I respectfully disagree. While I agree that it can be called as "multi-tenant", I am not sure if the implementation should be the addition of a new abstraction layer on top of the volumes as volumes are already created to provide very similar features. Therefore, (IMHO) we should consider to re-use our existing abstraction (volumes) as a solution for the multi-tenancy.

Volumes are independent of the account-namespace insolations, which is definitely a missing point. But based on the requirements, what we are looking for, is more like a user/group management feature ("tell me the user lists what I am managing..."), it's not a volume level user management.

My suggestions:

 * Investigate alternatives which can provide almost the same functionalities with significant less maintenance burden (and less implementation work !!!)
   * Using volume for bucket-namespace isolation instead of a new multi-tenancy abstraction layer
   * Using external group management / credential handling vs. investing in our user management (some investment may be required on our side, for making it more kerberos free, but it's an independent question) 
 * Compare the possible solutions with pro/con list (complexity, features, maintenance cost, ...)
 * Separate multi-tenancy and kerberos-independent secret management questions

Thank you again the answers.



> Multi-Tenant Support in Ozone
> -----------------------------
>
>                 Key: HDDS-4944
>                 URL: https://issues.apache.org/jira/browse/HDDS-4944
>             Project: Apache Ozone
>          Issue Type: New Feature
>          Components: Ozone CLI, Ozone Datanode, Ozone Manager, Ozone Recon, S3, SCM, Security
>    Affects Versions: 1.2.0
>            Reporter: Prashant Pogde
>            Assignee: Prashant Pogde
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: Apache-S3-compatible-Multi-Tenant-Ozone-short.pdf.gz, Ozone MultiTenant Feature _ Requirements and Abstractions-3.pdf, Ozone, Multi-tenancy, S3, Kerberos....pdf, UseCaseAWSCompatibility.pdf, UseCaseCephCompatibility.pdf, UseCaseConfigureMultiTenancy.png, UseCaseCurrentOzoneS3BackwardCompatibility.pdf, VariousActorsInteractions.png
>
>
> This Jira will be used to track a new feature for Multi-Tenant support in Ozone. Initially Multi-Tenant feature would be limited to ozone-users accessing Ozone over S3 interface.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org