You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by sh...@apache.org on 2018/09/18 02:06:14 UTC
[38/50] [abbrv] hadoop git commit: HDDS-435. Enhance the existing
ozone documentation. Contributed by Elek, Marton.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8af84535/hadoop-ozone/docs/content/Settings.md
----------------------------------------------------------------------
diff --git a/hadoop-ozone/docs/content/Settings.md b/hadoop-ozone/docs/content/Settings.md
new file mode 100644
index 0000000..bfa644e
--- /dev/null
+++ b/hadoop-ozone/docs/content/Settings.md
@@ -0,0 +1,139 @@
+---
+title: Configuration
+weight: 1
+menu:
+ main:
+ parent: Starting
+ weight: 2
+---
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+
+
+
+If you are feeling adventurous, you can setup ozone in a real cluster.
+Setting up a real cluster requires us to understand the components of Ozone.
+Ozone is designed to work concurrently with HDFS. However, Ozone is also
+capable of running independently. The components of ozone are the same in both approaches.
+
+## Ozone Components
+
+1. Ozone Manager - Is the server that is in charge of the namespace of Ozone. Ozone Manager is responsible for all volume, bucket and key operations.
+2. Storage Container Manager - Acts as the block manager. Ozone Manager
+requests blocks from SCM, to which clients can write data.
+3. Datanodes - Ozone data node code runs inside the HDFS datanode or in the independent deployment case runs an ozone datanode daemon.
+
+
+
+
+## Setting up an Ozone only cluster
+
+* Please untar the ozone-0.2.1-SNAPSHOT to the directory where you are going
+to run Ozone from. We need Ozone jars on all machines in the cluster. So you
+need to do this on all machines in the cluster.
+
+* Ozone relies on a configuration file called ```ozone-site.xml```. To
+generate a template that you can replace with proper values, please run the
+following command. This will generate a template called ```ozone-site.xml``` at
+the specified path (directory).
+
+{{< highlight bash >}}
+ozone genconf -output <path>
+{{< /highlight >}}
+
+Let us look at the settings inside the generated file (ozone-site.xml) and
+how they control ozone. Once the right values are defined, this file
+needs to be copied to ```ozone directory/etc/Hadoop```.
+
+
+* **ozone.enabled** This is the most critical setting for ozone.
+Ozone is a work in progress and users have to enable this service explicitly.
+By default, Ozone is disabled. Setting this flag to `true` enables ozone in the
+HDFS or Ozone cluster.
+
+Here is an example,
+
+{{< highlight xml >}}
+ <property>
+ <name>ozone.enabled</name>
+ <value>True</value>
+ </property>
+{{< /highlight >}}
+
+* **ozone.metadata.dirs** Allows Administrators to specify where the
+ metadata must reside. Usually you pick your fastest disk (SSD if
+ you have them on your nodes). OzoneManager, SCM and datanode will write the
+ metadata to this path. This is a required setting, if this is missing Ozone
+ will fail to come up.
+
+ Here is an example,
+
+{{< highlight xml >}}
+ <property>
+ <name>ozone.metadata.dirs</name>
+ <value>/data/disk1/meta</value>
+ </property>
+{{< /highlight >}}
+
+* **ozone.scm.names** Storage container manager(SCM) is a distributed block
+ service which is used by ozone. This property allows data nodes to discover
+ SCM's address. Data nodes send heartbeat to SCM.
+ Until HA feature is complete, we configure ozone.scm.names to be a
+ single machine.
+
+ Here is an example,
+
+ {{< highlight xml >}}
+ <property>
+ <name>ozone.scm.names</name>
+ <value>scm.hadoop.apache.org</value>
+ </property>
+ {{< /highlight >}}
+
+ * **ozone.scm.datanode.id** Data nodes generate a Unique ID called Datanode
+ ID. This identity is written to the file specified by this path. *Data nodes
+ will create this path if it doesn't exist already.*
+
+Here is an example,
+{{< highlight xml >}}
+ <property>
+ <name>ozone.scm.datanode.id</name>
+ <value>/data/disk1/meta/node/datanode.id</value>
+ </property>
+{{< /highlight >}}
+
+* **ozone.om.address** OM server address. This is used by OzoneClient and
+Ozone File System.
+
+Here is an example,
+{{< highlight xml >}}
+ <property>
+ <name>ozone.om.address</name>
+ <value>ozonemanager.hadoop.apache.org</value>
+ </property>
+{{< /highlight >}}
+
+
+### Ozone Settings Summary
+
+| Setting | Value | Comment |
+|--------------------------------|------------------------------|------------------------------------------------------------------|
+| ozone.enabled | true | This enables SCM and containers in HDFS cluster. |
+| ozone.metadata.dirs | file path | The metadata will be stored here. |
+| ozone.scm.names | SCM server name | Hostname:port or IP:port address of SCM. |
+| ozone.scm.block.client.address | SCM server name and port | Used by services like OM |
+| ozone.scm.client.address | SCM server name and port | Used by client-side |
+| ozone.scm.datanode.address | SCM server name and port | Used by datanode to talk to SCM |
+| ozone.om.address | OM server name | Used by Ozone handler and Ozone file system. |
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8af84535/hadoop-ozone/docs/content/VolumeCommands.md
----------------------------------------------------------------------
diff --git a/hadoop-ozone/docs/content/VolumeCommands.md b/hadoop-ozone/docs/content/VolumeCommands.md
new file mode 100644
index 0000000..45811c5
--- /dev/null
+++ b/hadoop-ozone/docs/content/VolumeCommands.md
@@ -0,0 +1,100 @@
+---
+title: Volume Commands
+menu:
+ main:
+ parent: Client
+ weight: 2
+---
+
+Volume commands generally need administrator privileges. The ozone shell supports the following volume commands.
+
+ * [create](#create)
+ * [delete](#delete)
+ * [info](#info)
+ * [list](#list)
+ * [update](#update)
+
+### Create
+
+The volume create command allows an administrator to create a volume and
+assign it to a user.
+
+***Params:***
+
+| Arguments | Comment |
+|--------------------------------|-----------------------------------------|
+| -q, --quota | Optional, This argument that specifies the maximum size this volume can use in the Ozone cluster. |
+| -u, --user | Required, The name of the user who owns this volume. This user can create, buckets and keys on this volume. |
+| Uri | The name of the volume. |
+
+{{< highlight bash >}}
+ozone sh volume create --quota=1TB --user=bilbo /hive
+{{< /highlight >}}
+
+The above command will create a volume called _hive_ on the ozone cluster. This
+volume has a quota of 1TB, and the owner is _bilbo_.
+
+### Delete
+
+The volume delete commands allows an administrator to delete a volume. If the
+volume is not empty then this command will fail.
+
+***Params:***
+
+| Arguments | Comment |
+|--------------------------------|-----------------------------------------|
+| Uri | The name of the volume.
+
+{{< highlight bash >}}
+ozone sh volume delete /hive
+{{< /highlight >}}
+
+The above command will delete the volume hive, if the volume has no buckets
+inside it.
+
+### Info
+
+The volume info commands returns the information about the volume including
+quota and owner information.
+***Params:***
+
+| Arguments | Comment |
+|--------------------------------|-----------------------------------------|
+| Uri | The name of the volume.
+
+{{< highlight bash >}}
+ozone sh volume info /hive
+{{< /highlight >}}
+
+The above command will print out the information about hive volume.
+
+### List
+
+The volume list command will list the volumes owned by a user.
+
+{{< highlight bash >}}
+ozone sh volume list --user hadoop
+{{< /highlight >}}
+
+The above command will print out all the volumes owned by the user hadoop.
+
+### Update
+
+The volume update command allows changing of owner and quota on a given volume.
+
+***Params:***
+
+| Arguments | Comment |
+|--------------------------------|-----------------------------------------|
+| -q, --quota | Optional, This argument that specifies the maximum size this volume can use in the Ozone cluster. |
+| -u, --user | Optional, The name of the user who owns this volume. This user can create, buckets and keys on this volume. |
+| Uri | The name of the volume. |
+
+{{< highlight bash >}}
+ozone sh volume update --quota=10TB /hive
+{{< /highlight >}}
+
+The above command updates the volume quota to 10TB.
+
+You can try out these commands from the docker instance of the [Alpha
+Cluster](runningviadocker.html).
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8af84535/hadoop-ozone/docs/content/_index.md
----------------------------------------------------------------------
diff --git a/hadoop-ozone/docs/content/_index.md b/hadoop-ozone/docs/content/_index.md
index 383b2e0..19340bf 100644
--- a/hadoop-ozone/docs/content/_index.md
+++ b/hadoop-ozone/docs/content/_index.md
@@ -17,86 +17,20 @@ weight: -10
limitations under the License. See accompanying LICENSE file.
-->
-
Ozone is an Object store for Apache Hadoop. It aims to scale to billions of
-keys.
The following is a high-level overview of the core components of Ozone.
+# Apache Hadoop Ozone
-
+Ozone is a scalable, distributed object store for Hadoop. Applications like
+Apache Spark, Hive and YARN, can run against Ozone without any
+modifications. Ozone comes with a [Java client library]({{< ref "JavaApi.md"
+>}}) and a [command line interface] ({{< ref "CommandShell.md#shell" >}}) which makes it easy to use Ozone. This client library supports both RPC and REST protocols.
-The main elements of Ozone are
:
+Ozone consists of volumes, buckets, and Keys.
-## Clients
+* Volumes are similar to user accounts. Only administrators can create or delete volumes.
+* Buckets are similar to directories. A bucket can contain any number of keys, but buckets cannot contain other buckets.
+* Keys are similar to files. A bucket can contain any number of keys.
-Ozone ships with a set of ready-made clients. They are
Ozone CLI and Freon.
- * [Ozone CLI](./OzoneCommandShell.html) is the command line interface like 'hdfs' command.
- * Freon is a load generation tool for Ozone.
+<a href="{{< ref "RunningViaDocker.md" >}}"><button class="btn btn-danger btn-lg">Getting started</button></a>
-## REST Handler
-
-Ozone provides both an RPC (Remote Procedure Call) as well as a REST
-(Representational State Transfer) style interface. This allows clients to be
-written in many languages quickly. Ozone strives to maintain a similar
-interface between REST and RPC. The Rest handler offers the REST protocol
-services of Ozone.
-
-For most purposes, a client can make one line change to switch from REST to
-RPC or vice versa.
-
-## Ozone File System
-
-Ozone file system (TODO: Add documentation) is a Hadoop compatible file system.
-This is the important user-visible component of ozone.
-This allows Hadoop services and applications like Hive/Spark to run against
-Ozone without any change.
-
-## Ozone Client
-
-This is like DFSClient in HDFS. This acts as the standard client to talk to
-Ozone. All other components that we have discussed so far rely on Ozone client
-(TODO: Add Ozone client documentation).
-
-## Ozone Manager
-
-Ozone Manager (OM) takes care of the Ozone's namespace.
-All ozone entities like volumes, buckets and keys are managed by OM
-(TODO: Add OM documentation). In short, OM is the metadata manager for Ozone.
-OM talks to blockManager(SCM) to get blocks and passes it on to the Ozone
-client. Ozone client writes data to these blocks.
-OM will eventually be replicated via Apache Ratis for High Availability.
-
-## Storage Container Manager
-Storage Container Manager (SCM) is the block and cluster manager for Ozone.
-SCM along with data nodes offer a service called 'containers'.
-A container is a group unrelated of blocks that are managed together
-as a single entity.
-
-SCM offers the following abstractions.
-
-
-
-### Blocks
-
-Blocks are like blocks in HDFS. They are replicated store of data.
-
-### Containers
-
-A collection of blocks replicated and managed together.
-
-### Pipelines
-
-SCM allows each container to choose its method of replication.
-For example, a container might decide that it needs only one copy of a block
-and might choose a stand-alone pipeline. Another container might want to have
-a very high level of reliability and pick a RATIS based pipeline. In other
-words, SCM allows different kinds of replication strategies to co-exist.
-
-### Pools
-
-A group of data nodes is called a pool. For scaling purposes,
-we define a pool as a set of machines. This makes management of datanodes
-easier.
-
-### Nodes
-
-The data node where data is stored.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8af84535/hadoop-ozone/docs/pom.xml
----------------------------------------------------------------------
diff --git a/hadoop-ozone/docs/pom.xml b/hadoop-ozone/docs/pom.xml
index 92680ad..d8edd15 100644
--- a/hadoop-ozone/docs/pom.xml
+++ b/hadoop-ozone/docs/pom.xml
@@ -47,10 +47,6 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd">
</executions>
<configuration>
<executable>dev-support/bin/generate-site.sh</executable>
- <arguments>
- <argument>-b</argument>
- <argument>/docs</argument>
- </arguments>
</configuration>
</plugin>
<plugin>
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8af84535/hadoop-ozone/docs/themes/ozonedoc/layouts/_default/single.html
----------------------------------------------------------------------
diff --git a/hadoop-ozone/docs/themes/ozonedoc/layouts/_default/single.html b/hadoop-ozone/docs/themes/ozonedoc/layouts/_default/single.html
index 0fdd1ba..1970f73 100644
--- a/hadoop-ozone/docs/themes/ozonedoc/layouts/_default/single.html
+++ b/hadoop-ozone/docs/themes/ozonedoc/layouts/_default/single.html
@@ -21,7 +21,10 @@
<div class="row">
{{ partial "sidebar.html" . }}
<div class="col-sm-9 col-sm-offset-3 col-md-10 col-md-offset-2 main">
+ <h1>{{ .Title }}</h1>
+ <div class="col-md-9">
{{ .Content }}
+ </div>
</div>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8af84535/hadoop-ozone/docs/themes/ozonedoc/layouts/partials/header.html
----------------------------------------------------------------------
diff --git a/hadoop-ozone/docs/themes/ozonedoc/layouts/partials/header.html b/hadoop-ozone/docs/themes/ozonedoc/layouts/partials/header.html
index c1f47a9..220db57 100644
--- a/hadoop-ozone/docs/themes/ozonedoc/layouts/partials/header.html
+++ b/hadoop-ozone/docs/themes/ozonedoc/layouts/partials/header.html
@@ -23,9 +23,9 @@
<title>Documentation for Apache Hadoop Ozone</title>
<!-- Bootstrap core CSS -->
- <link href="{{ "css/bootstrap.min.css" | relURL }}" rel="stylesheet">
+ <link href="css/bootstrap.min.css" rel="stylesheet">
<!-- Custom styles for this template -->
- <link href="{{ "css/ozonedoc.css" | relURL }}" rel="stylesheet">
+ <link href="css/ozonedoc.css" rel="stylesheet">
</head>
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8af84535/hadoop-ozone/docs/themes/ozonedoc/layouts/partials/sidebar.html
----------------------------------------------------------------------
diff --git a/hadoop-ozone/docs/themes/ozonedoc/layouts/partials/sidebar.html b/hadoop-ozone/docs/themes/ozonedoc/layouts/partials/sidebar.html
index b043911..36bed1a 100644
--- a/hadoop-ozone/docs/themes/ozonedoc/layouts/partials/sidebar.html
+++ b/hadoop-ozone/docs/themes/ozonedoc/layouts/partials/sidebar.html
@@ -12,6 +12,7 @@
limitations under the License. See accompanying LICENSE file.
-->
<div class="col-sm-3 col-md-2 sidebar">
+ <img src="ozone-logo.png" style="max-width: 100%;"/>
<ul class="nav nav-sidebar">
{{ $currentPage := . }}
{{ range .Site.Menus.main }}
@@ -31,7 +32,12 @@
</li>
{{ else }}
<li class="{{ if $currentPage.IsMenuCurrent "main" . }}active{{ end }}">
- <a href="{{ .URL }}">
+ {{ if eq .URL "/" }}
+ <a href="index.html">
+ {{ else }}
+ <a href="{{ .URL }}">
+ {{ end }}
+
{{ .Pre }}
<span>{{ .Name }}</span>
</a>
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8af84535/hadoop-ozone/docs/themes/ozonedoc/static/css/ozonedoc.css
----------------------------------------------------------------------
diff --git a/hadoop-ozone/docs/themes/ozonedoc/static/css/ozonedoc.css b/hadoop-ozone/docs/themes/ozonedoc/static/css/ozonedoc.css
index 39fae72..7f68b64 100644
--- a/hadoop-ozone/docs/themes/ozonedoc/static/css/ozonedoc.css
+++ b/hadoop-ozone/docs/themes/ozonedoc/static/css/ozonedoc.css
@@ -23,10 +23,20 @@
/* Move down content because we have a fixed navbar that is 50px tall */
body {
padding-top: 50px;
- font-size: 150%;
+ font-size: 14pt;
}
+pre {
+ font-size: 14pt;
+}
+
+a {
+ color: #448628;
+}
+a:hover {
+ color: #245F0B;
+}
/*
* Global add-ons
*/
@@ -86,7 +96,7 @@ body {
.nav-sidebar .active > a:hover,
.nav-sidebar .active > a:focus {
color: #fff;
- background-color: #428bca;
+ background-color: #73B148;
}
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org