You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Anu Engineer (JIRA)" <ji...@apache.org> on 2018/10/04 19:36:00 UTC
[jira] [Comment Edited] (HDDS-572) Support S3 buckets as first
class objects in Ozone Manager - 1
[ https://issues.apache.org/jira/browse/HDDS-572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16638738#comment-16638738 ]
Anu Engineer edited comment on HDDS-572 at 10/4/18 7:35 PM:
------------------------------------------------------------
What this solves is not the virtual host problem. What this intends to solve is the S3Gateway user needing to know what ozone volume the bucket is in.
Let me explain in detail:
1. In the current model, if a user wants to create an S3 bucket, they must first create an ozone volume and pass that in the endpoint URL. In other words, they cannot just use AWS CLI only. The user must first do an ozone operation.
2. Whenever a user accesses the s3 bucket, the user must know the volume name. This Jira proposes to build an API to discover this information from an s3 bucket.
3. Consistency in behavior:
3.1 When the user does a list buckets operation (GET SERVICE) they expect to see all buckets owned by them. With the current approach of endPoint URL, we cannot achieve that. This is the most important reason for supporting S3 buckets as a first-class object in OzoneManager.
3.2 When you create buckets in S3, if a bucket name already exists, it will return an error. With the volume-based approach, we can do that only for a specific volume. So the users can be easily confused, that the create bucket works, but the buckets are not the same. This violates some of the core semantics of S3. With this patch, we address that.
4. Discovery: Currently, we rely on users to remember what the S3 bucket name is and an additional piece of information where the Ozone bucket is. With this patch, they need to remember what they remember for S3. The cluster and bucket, the ozone bucket name can be retrieved as needed.
5. Ease of use: When someone wants to mount an S3 bucket via Ozone FS, they can ask the server what is the mount path. Ozone Manager will return the path.
6. Adding security credentials: Eventually we will need to support S3 security model. So we need someplace where we store that creds, this patch makes that also easy.
In the future patches, I will expose this functionality to Ozone RPC client and will follow up with some cache for the S3Gateway. I have also discussed this issue at a high level in the design doc posted to this JIRA.
was (Author: anu):
What this solves is not the virtual host problem. What this intends to solve is the S3Gateway user needing to know what ozone volume the bucket is in.
Let me explain in detail:
1. In the current model, if a user wants to create an S3 bucket, they must first create an ozone volume and pass that in the endpoint URL. In other words, they cannot just use AWS CLI only. The user must first do an ozone operation.
2. Whenever a user accesses the s3 bucket, the user must know the volume name. This Jira proposes to build an API to discover this information from an s3 bucket.
3. Consistency in behavior:
3.1 When the user does a list buckets operation (GET SERVICE) they expect to see all buckets owned by them. With the current approach of endPoint URL, we cannot achieve that. This is the most important reason for supporting S3 buckets as a first-class object in OzoneManager.
3.2 When you create buckets in S3, if a bucket name already exists, it will return an error. With the volume-based approach, we can do that only for a specific volume. So the users can be easily confused, if that the create bucket works, but the buckets are not the same. This violates some of the core semantics of S3. With this patch, we address that.
4. Discovery: Currently, we rely on users to remember what the S3 bucket name is and an additional piece of information where the Ozone bucket is. With this patch, they need to remember what they remember for S3. The cluster and bucket, the ozone bucket name can be retrieved as needed.
5. Ease of use: When someone wants to mount an S3 bucket via Ozone FS, they can ask the server what is the mount path. Ozone Manager will return the path.
6. Adding security credentials: Eventually we will need to support S3 security model. So we need someplace where we store that creds, this patch makes that also easy.
In the future patches, I will expose this functionality to Ozone RPC client and will follow up with some cache for the S3Gateway. I have also discussed this issue at a high level in the design doc posted to this JIRA.
> Support S3 buckets as first class objects in Ozone Manager - 1
> --------------------------------------------------------------
>
> Key: HDDS-572
> URL: https://issues.apache.org/jira/browse/HDDS-572
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Reporter: Anu Engineer
> Assignee: Anu Engineer
> Priority: Major
> Attachments: HDDS-572.001.patch
>
>
> This Jira proposes to add support for S3 buckets as first class objects in Ozone Manager. Currently we take the Ozone volume via the endPoint URL in AWS sdk. With this(and the next 2 patchs), we can move away from using ozone volume in the URL.
> cc: [~elek], [~bharatviswa]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org