You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by bh...@apache.org on 2018/10/31 18:48:46 UTC
hadoop git commit: HDDS-677. Create documentation for s3 gateway to
the docs. Contributed by Elek Marton.
Repository: hadoop
Updated Branches:
refs/heads/trunk 08bb0362e -> 6668c19da
HDDS-677. Create documentation for s3 gateway to the docs. Contributed by Elek Marton.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/6668c19d
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/6668c19d
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/6668c19d
Branch: refs/heads/trunk
Commit: 6668c19dafde530c43ccacb23de11455ae1813b5
Parents: 08bb036
Author: Bharat Viswanadham <bh...@apache.org>
Authored: Wed Oct 31 11:48:23 2018 -0700
Committer: Bharat Viswanadham <bh...@apache.org>
Committed: Wed Oct 31 11:48:23 2018 -0700
----------------------------------------------------------------------
hadoop-ozone/docs/content/S3.md | 130 +++++++++++++++++++++++++++++++++++
1 file changed, 130 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/6668c19d/hadoop-ozone/docs/content/S3.md
----------------------------------------------------------------------
diff --git a/hadoop-ozone/docs/content/S3.md b/hadoop-ozone/docs/content/S3.md
new file mode 100644
index 0000000..cfefaf4
--- /dev/null
+++ b/hadoop-ozone/docs/content/S3.md
@@ -0,0 +1,130 @@
+---
+title: S3
+menu:
+ main:
+ parent: Client
+ weight: 1
+---
+
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+Ozone provides S3 compatible REST interface to use the object store data with any S3 compatible tools.
+
+## Getting started
+
+S3 Gateway is a separated component which provides the S3 compatible. It should be started additional to the regular Ozone components.
+
+You can start a docker based cluster, including the S3 gateway from the release package.
+
+Go to the `compose/ozones3` directory, and start the server:
+
+```bash
+docker-compose up -d
+```
+
+You can access the S3 gateway at `http://localhost:9878`
+
+## URL Schema
+
+Ozone S3 gateway supports both the virtual-host-style URL s3 bucket addresses (eg. http://bucketname.host:9878) and the path-style addresses (eg. http://host:9878/bucketname)
+
+By default it uses the path-style addressing. To use virtual host style URLs set your main domain name in your `ozone-site.xml`:
+
+```xml
+<property>
+ <name>ozone.s3g.domain.name</name>
+ <value>s3g.internal</value>
+</property>
+```
+
+## Bucket browser
+
+Bucket's could be browsed from the browser with adding `?browser=true` to the bucket URL.
+
+For example the content of the 'testbucket' could be checked from the browser using the URL http://localhost:9878/testbucket?browser=true
+
+
+## Implemented REST endpoints
+
+Operations on S3Gateway service:
+
+Endpoint | Status |
+------------|-------------|
+GET service | implemented |
+
+Operations on Bucket:
+
+Endpoint | Status | Notes
+------------------------------------|-------------|---------------
+GET Bucket (List Objects) Version 2 | implemented |
+HEAD Bucket | implemented |
+DELETE Bucket | implemented |
+PUT Bucket (Create bucket) | implemented |
+Delete Multiple Objects (POST) | implemented |
+
+Operation on Objects:
+
+Endpoint | Status | Notes
+------------------------------------|-----------------|---------------
+PUT Object | implemented |
+GET Object | implemented | Range headers are not supported
+Multipart Uplad | not implemented |
+DELETE Object | implemented |
+HEAD Object | implemented |
+
+
+## Security
+
+Security is not yet implemented, you can *use* any AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
+
+Note: Ozone has a notion for 'volumes' which is missing from the S3 Rest endpoint. Under the hood S3 bucket names are maped to Ozone 'volume/bucket' locations (depending from the given authentication information).
+
+To show the storage location of a S3 bucket, use the `ozone sh bucket path <bucketname>` command.
+
+```
+aws s3api --endpoint-url http://localhost:9878 create-bucket --bucket=bucket1
+
+ozone sh bucket path bucket1
+Volume name for S3Bucket is : s3thisisakey
+Ozone FileSystem Uri is : o3fs://bucket1.s3thisisakey
+```
+
+## Clients
+
+### AWS Cli
+
+`aws` CLI could be used with specifying the custom REST endpoint.
+
+```
+aws s3api --endpoint http://localhost:9878 create-bucket --bucket buckettest
+```
+
+Or
+
+```
+aws s3 ls --endpoint http://localhost:9878 s3://buckettest
+```
+
+### S3 Fuse driver (goofys)
+
+Goofys is a S3 FUSE driver. It could be used to any mount any Ozone bucket as posix file system:
+
+
+```
+goofys --endpoint http://localhost:9878 bucket1 /mount/bucket1
+```
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org