You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@knox.apache.org by km...@apache.org on 2013/03/20 17:06:12 UTC
svn commit: r1458909 - in /incubator/knox/site: examples.html
getting-started.html release-0-2-0.html
Author: kminder
Date: Wed Mar 20 16:06:11 2013
New Revision: 1458909
URL: http://svn.apache.org/r1458909
Log:
Add files that were missed with svn add.
Added:
incubator/knox/site/examples.html
incubator/knox/site/getting-started.html
incubator/knox/site/release-0-2-0.html
Added: incubator/knox/site/examples.html
URL: http://svn.apache.org/viewvc/incubator/knox/site/examples.html?rev=1458909&view=auto
==============================================================================
--- incubator/knox/site/examples.html (added)
+++ incubator/knox/site/examples.html Wed Mar 20 16:06:11 2013
@@ -0,0 +1,289 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<!-- Generated by Apache Maven Doxia at Mar 20, 2013 -->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
+ <title></title>
+ <style type="text/css" media="all">
+ @import url("./css/maven-base.css");
+ @import url("./css/maven-theme.css");
+ @import url("./css/site.css");
+ </style>
+ <link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
+ <meta name="Date-Revision-yyyymmdd" content="20130320" />
+ <meta http-equiv="Content-Language" content="en" />
+
+ </head>
+ <body class="composite">
+ <div id="banner">
+ <a href="./" id="bannerLeft">
+ <img src="images/apache-logo.gif" alt="Knox Gateway" />
+ </a>
+ <a href=".." id="bannerRight">
+ <img src="images/apache-incubator-logo.png" alt="Apache Incubator" />
+ </a>
+ <div class="clear">
+ <hr/>
+ </div>
+ </div>
+ <div id="breadcrumbs">
+
+
+ <div class="xleft">
+ <a href="index.html" title="Knox">Knox</a>
+ >
+
+ </div>
+ <div class="xright"> <a href="https://git-wip-us.apache.org/repos/asf/incubator-knox.git" class="externalLink" title="Git">Git</a>
+ |
+ <a href="https://svn.apache.org/repos/asf/incubator/knox/site" class="externalLink" title="Svn">Svn</a>
+ |
+ <a href="https://issues.apache.org/jira/browse/KNOX" class="externalLink" title="Jira">Jira</a>
+ |
+ <a href="https://cwiki.apache.org/confluence/display/KNOX/Knox" class="externalLink" title="Wiki">Wiki</a>
+
+
+ | <span id="publishDate">Last Published: 2013-03-20</span>
+ | <span id="projectVersion">Version: 0.2.0-SNAPSHOT</span>
+ </div>
+ <div class="clear">
+ <hr/>
+ </div>
+ </div>
+ <div id="leftColumn">
+ <div id="navcolumn">
+
+
+ <h5>Knox</h5>
+ <ul>
+ <li class="none">
+ <a href="index.html" title="Welcome">Welcome</a>
+ </li>
+ <li class="none">
+ <a href="news.html" title="News">News</a>
+ </li>
+ <li class="none">
+ <a href="license.html" title="License">License</a>
+ </li>
+ </ul>
+ <h5>Documentation</h5>
+ <ul>
+ <li class="none">
+ <a href="getting-started.html" title="Getting Started">Getting Started</a>
+ </li>
+ <li class="none">
+ <strong>Usage Examples</strong>
+ </li>
+ <li class="none">
+ <a href="client.html" title="Client (KnoxShell DSL)">Client (KnoxShell DSL)</a>
+ </li>
+ <li class="none">
+ <a href="https://cwiki.apache.org/confluence/display/KNOX/Knox" class="externalLink" title="Wiki">Wiki</a>
+ </li>
+ </ul>
+ <h5>Releases</h5>
+ <ul>
+ <li class="none">
+ <a href="release-0-2-0.html" title="0.2.0 (coming soon)">0.2.0 (coming soon)</a>
+ </li>
+ <li class="none">
+ <a href="roadmap-0-3-0.html" title="0.3.0 (planning)">0.3.0 (planning)</a>
+ </li>
+ </ul>
+ <h5>Processes</h5>
+ <ul>
+ <li class="none">
+ <a href="build-process.html" title="Build">Build</a>
+ </li>
+ <li class="none">
+ <a href="release-process.html" title="Release">Release</a>
+ </li>
+ <li class="none">
+ <a href="contribute-process.html" title="Contribute">Contribute</a>
+ </li>
+ <li class="none">
+ <a href="site-process.html" title="Site Publication">Site Publication</a>
+ </li>
+ </ul>
+ <h5>Resources</h5>
+ <ul>
+ <li class="none">
+ <a href="https://git-wip-us.apache.org/repos/asf/incubator-knox.git" class="externalLink" title="Source Code">Source Code</a>
+ </li>
+ <li class="none">
+ <a href="team-list.html" title="Project Team">Project Team</a>
+ </li>
+ <li class="none">
+ <a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a>
+ </li>
+ <li class="none">
+ <a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a>
+ </li>
+ </ul>
+ <h5>ASF</h5>
+ <ul>
+ <li class="none">
+ <a href="http://www.apache.org/foundation/how-it-works.html" class="externalLink" title="How Apache Works">How Apache Works</a>
+ </li>
+ <li class="none">
+ <a href="http://www.apache.org/foundation/" class="externalLink" title="Foundation">Foundation</a>
+ </li>
+ <li class="none">
+ <a href="http://www.apache.org/foundation/sponsorship.html" class="externalLink" title="Sponsoring Apache">Sponsoring Apache</a>
+ </li>
+ <li class="none">
+ <a href="http://www.apache.org/foundation/thanks.html" class="externalLink" title="Thanks">Thanks</a>
+ </li>
+ </ul>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="poweredBy" alt="Built by Maven" src="./images/logos/maven-feather.png" />
+ </a>
+
+
+ </div>
+ </div>
+ <div id="bodyColumn">
+ <div id="contentBox">
+ <!-- -
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License. --><p></p><div class="section"><h2>Example #3: WebHDFS & Templeton/WebHCat<a name="Example_3:_WebHDFS__TempletonWebHCat"></a></h2><p>The example below illustrates the sequence of curl commands that could be used to run a word count map reduce job. It utilizes the hadoop-examples.jar from a Hadoop install for running a simple word count job. Take care to follow the instructions below for steps 4/5 and 6/7 where the Location header returned by the call to the NameNode is copied for use with the call to the DataNode that follows it.</p>
+<div class="source"><pre># 0. Optionally cleanup the test directory in case a previous example was run without cleaning up.
+curl -i -k -u mapred:mapred-password -X DELETE \
+ 'https://localhost:8443/gateway/sample/namenode/api/v1/tmp/test?op=DELETE&recursive=true'
+
+# 1. Create a test input directory /tmp/test/input
+curl -i -k -u mapred:mapred-password -X PUT \
+ 'https://localhost:8443/gateway/sample/namenode/api/v1/tmp/test/input?op=MKDIRS'
+
+# 2. Create a test output directory /tmp/test/input
+curl -i -k -u mapred:mapred-password -X PUT \
+ 'https://localhost:8443/gateway/sample/namenode/api/v1/tmp/test/output?op=MKDIRS'
+
+# 3. Create the inode for hadoop-examples.jar in /tmp/test
+curl -i -k -u mapred:mapred-password -X PUT \
+ 'https://localhost:8443/gateway/sample/namenode/api/v1/tmp/test/hadoop-examples.jar?op=CREATE'
+
+# 4. Upload hadoop-examples.jar to /tmp/test. Use a hadoop-examples.jar from a Hadoop install.
+curl -i -k -u mapred:mapred-password -T hadoop-examples.jar -X PUT '{Value Location header from command above}'
+
+# 5. Create the inode for a sample file README in /tmp/test/input
+curl -i -k -u mapred:mapred-password -X PUT \
+ 'https://localhost:8443/gateway/sample/namenode/api/v1/tmp/test/input/README?op=CREATE'
+
+# 6. Upload readme.txt to /tmp/test/input. Use the readme.txt in {GATEWAY_HOME}.
+curl -i -k -u mapred:mapred-password -T README -X PUT '{Value of Location header from command above}'
+
+# 7. Submit the word count job via WebHCat/Templeton.
+# Take note of the Job ID in the JSON response as this will be used in the next step.
+curl -v -i -k -u mapred:mapred-password -X POST \
+ -d jar=/tmp/test/hadoop-examples.jar -d class=wordcount \
+ -d arg=/tmp/test/input -d arg=/tmp/test/output \
+ 'https://localhost:8443/gateway/sample/templeton/api/v1/mapreduce/jar'
+
+# 8. Look at the status of the job
+curl -i -k -u mapred:mapred-password -X GET \
+ 'https://localhost:8443/gateway/sample/templeton/api/v1/queue/{Job ID returned in JSON body from previous step}'
+
+# 9. Look at the status of the job queue
+curl -i -k -u mapred:mapred-password -X GET \
+ 'https://localhost:8443/gateway/sample/templeton/api/v1/queue'
+
+# 10. List the contents of the output directory /tmp/test/output
+curl -i -k -u mapred:mapred-password -X GET \
+ 'https://localhost:8443/gateway/sample/namenode/api/v1/tmp/test/output?op=LISTSTATUS'
+
+# 11. Optionally cleanup the test directory
+curl -i -k -u mapred:mapred-password -X DELETE \
+ 'https://localhost:8443/gateway/sample/namenode/api/v1/tmp/test?op=DELETE&recursive=true'
+</pre></div><p></p></div><div class="section"><h2>Example #4: WebHDFS & Oozie<a name="Example_4:_WebHDFS__Oozie"></a></h2><p>The example below illustrates the sequence of curl commands that could be used to run a word count map reduce job via an Oozie workflow. It utilizes the hadoop-examples.jar from a Hadoop install for running a simple word count job. Take care to follow the instructions below where replacement values are required. These replacement values are identivied with { } markup.</p>
+<div class="source"><pre># 0. Optionally cleanup the test directory in case a previous example was run without cleaning up.
+curl -i -k -u mapred:mapred-password -X DELETE \
+ 'https://localhost:8443/gateway/sample/namenode/api/v1/tmp/test?op=DELETE&recursive=true'
+
+# 1. Create the inode for workflow definition file in /tmp/test
+curl -i -k -u mapred:mapred-password -X PUT \
+ 'https://localhost:8443/gateway/sample/namenode/api/v1/tmp/test/workflow.xml?op=CREATE'
+
+# 2. Upload the workflow definition file. This file can be found in {GATEWAY_HOME}/templates
+curl -i -k -u mapred:mapred-password -T templates/workflow-definition.xml -X PUT \
+ '{Value Location header from command above}'
+
+# 3. Create the inode for hadoop-examples.jar in /tmp/test/lib
+curl -i -k -u mapred:mapred-password -X PUT \
+ 'https://localhost:8443/gateway/sample/namenode/api/v1/tmp/test/lib/hadoop-examples.jar?op=CREATE'
+
+# 4. Upload hadoop-examples.jar to /tmp/test/lib. Use a hadoop-examples.jar from a Hadoop install.
+curl -i -k -u mapred:mapred-password -T hadoop-examples.jar -X PUT \
+ '{Value Location header from command above}'
+
+# 5. Create the inode for a sample input file readme.txt in /tmp/test/input.
+curl -i -k -u mapred:mapred-password -X PUT \
+ 'https://localhost:8443/gateway/sample/namenode/api/v1/tmp/test/input/README?op=CREATE'
+
+# 6. Upload readme.txt to /tmp/test/input. Use the readme.txt in {GATEWAY_HOME}.
+# The sample below uses this README file found in {GATEWAY_HOME}.
+curl -i -k -u mapred:mapred-password -T README -X PUT \
+ '{Value of Location header from command above}'
+
+# 7. Create the job configuration file by replacing the {NameNode host:port} and {JobTracker host:port}
+# in the command below to values that match your Hadoop configuration.
+# NOTE: The hostnames must be resolvable by the Oozie daemon. The ports are the RPC ports not the HTTP ports.
+# For example {NameNode host:port} might be sandbox:8020 and {JobTracker host:port} sandbox:50300
+# The source workflow-configuration.xml file can be found in {GATEWAY_HOME}/templates
+# Alternatively, this file can copied and edited manually for environments without the sed utility.
+sed -e s/REPLACE.NAMENODE.RPCHOSTPORT/{NameNode host:port}/ \
+ -e s/REPLACE.JOBTRACKER.RPCHOSTPORT/{JobTracker host:port}/ \
+ <templates/workflow-configuration.xml >workflow-configuration.xml
+
+# 8. Submit the job via Oozie
+# Take note of the Job ID in the JSON response as this will be used in the next step.
+curl -i -k -u mapred:mapred-password -T workflow-configuration.xml -H Content-Type:application/xml -X POST \
+ 'https://localhost:8443/gateway/oozie/sample/api/v1/jobs?action=start'
+
+# 9. Query the job status via Oozie.
+curl -i -k -u mapred:mapred-password -X GET \
+ 'https://localhost:8443/gateway/sample/oozie/api/v1/job/{Job ID returned in JSON body from previous step}'
+
+# 10. List the contents of the output directory /tmp/test/output
+curl -i -k -u mapred:mapred-password -X GET \
+ 'https://localhost:8443/gateway/sample/namenode/api/v1/tmp/test/output?op=LISTSTATUS'
+
+# 11. Optionally cleanup the test directory
+curl -i -k -u mapred:mapred-password -X DELETE \
+ 'https://localhost:8443/gateway/sample/namenode/api/v1/tmp/test?op=DELETE&recursive=true'
+</pre></div><p></p></div><div class="section"><h2>Disclaimer<a name="Disclaimer"></a></h2><p>The Apache Knox Gateway is an effort undergoing incubation at the Apache Software Foundation (ASF), sponsored by the Apache Incubator PMC.</p><p>Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects.</p><p>While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.</p></div>
+ </div>
+ </div>
+ <div class="clear">
+ <hr/>
+ </div>
+ <div id="footer">
+ <div class="xright">
+ <?xml version="1.0" encoding="UTF-8"?>
+<footer>
+ <div class="row span12">Apache Knox Gateway, Apache, the Apache feather logo and the Apache Knox Gateway project logos
+ are trademarks of The Apache Software Foundation.
+ All other marks mentioned may be trademarks or registered trademarks of their respective owners.</div>
+ <div class="row span12">
+ <a href="privacy-policy.html">Privacy Policy</a>
+ </div>
+</footer>
+ </div>
+ <div class="clear">
+ <hr/>
+ </div>
+ </div>
+ </body>
+</html>
Added: incubator/knox/site/getting-started.html
URL: http://svn.apache.org/viewvc/incubator/knox/site/getting-started.html?rev=1458909&view=auto
==============================================================================
--- incubator/knox/site/getting-started.html (added)
+++ incubator/knox/site/getting-started.html Wed Mar 20 16:06:11 2013
@@ -0,0 +1,260 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<!-- Generated by Apache Maven Doxia at Mar 20, 2013 -->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
+ <title></title>
+ <style type="text/css" media="all">
+ @import url("./css/maven-base.css");
+ @import url("./css/maven-theme.css");
+ @import url("./css/site.css");
+ </style>
+ <link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
+ <meta name="Date-Revision-yyyymmdd" content="20130320" />
+ <meta http-equiv="Content-Language" content="en" />
+
+ </head>
+ <body class="composite">
+ <div id="banner">
+ <a href="./" id="bannerLeft">
+ <img src="images/apache-logo.gif" alt="Knox Gateway" />
+ </a>
+ <a href=".." id="bannerRight">
+ <img src="images/apache-incubator-logo.png" alt="Apache Incubator" />
+ </a>
+ <div class="clear">
+ <hr/>
+ </div>
+ </div>
+ <div id="breadcrumbs">
+
+
+ <div class="xleft">
+ <a href="index.html" title="Knox">Knox</a>
+ >
+
+ </div>
+ <div class="xright"> <a href="https://git-wip-us.apache.org/repos/asf/incubator-knox.git" class="externalLink" title="Git">Git</a>
+ |
+ <a href="https://svn.apache.org/repos/asf/incubator/knox/site" class="externalLink" title="Svn">Svn</a>
+ |
+ <a href="https://issues.apache.org/jira/browse/KNOX" class="externalLink" title="Jira">Jira</a>
+ |
+ <a href="https://cwiki.apache.org/confluence/display/KNOX/Knox" class="externalLink" title="Wiki">Wiki</a>
+
+
+ | <span id="publishDate">Last Published: 2013-03-20</span>
+ | <span id="projectVersion">Version: 0.2.0-SNAPSHOT</span>
+ </div>
+ <div class="clear">
+ <hr/>
+ </div>
+ </div>
+ <div id="leftColumn">
+ <div id="navcolumn">
+
+
+ <h5>Knox</h5>
+ <ul>
+ <li class="none">
+ <a href="index.html" title="Welcome">Welcome</a>
+ </li>
+ <li class="none">
+ <a href="news.html" title="News">News</a>
+ </li>
+ <li class="none">
+ <a href="license.html" title="License">License</a>
+ </li>
+ </ul>
+ <h5>Documentation</h5>
+ <ul>
+ <li class="none">
+ <strong>Getting Started</strong>
+ </li>
+ <li class="none">
+ <a href="examples.html" title="Usage Examples">Usage Examples</a>
+ </li>
+ <li class="none">
+ <a href="client.html" title="Client (KnoxShell DSL)">Client (KnoxShell DSL)</a>
+ </li>
+ <li class="none">
+ <a href="https://cwiki.apache.org/confluence/display/KNOX/Knox" class="externalLink" title="Wiki">Wiki</a>
+ </li>
+ </ul>
+ <h5>Releases</h5>
+ <ul>
+ <li class="none">
+ <a href="release-0-2-0.html" title="0.2.0 (coming soon)">0.2.0 (coming soon)</a>
+ </li>
+ <li class="none">
+ <a href="roadmap-0-3-0.html" title="0.3.0 (planning)">0.3.0 (planning)</a>
+ </li>
+ </ul>
+ <h5>Processes</h5>
+ <ul>
+ <li class="none">
+ <a href="build-process.html" title="Build">Build</a>
+ </li>
+ <li class="none">
+ <a href="release-process.html" title="Release">Release</a>
+ </li>
+ <li class="none">
+ <a href="contribute-process.html" title="Contribute">Contribute</a>
+ </li>
+ <li class="none">
+ <a href="site-process.html" title="Site Publication">Site Publication</a>
+ </li>
+ </ul>
+ <h5>Resources</h5>
+ <ul>
+ <li class="none">
+ <a href="https://git-wip-us.apache.org/repos/asf/incubator-knox.git" class="externalLink" title="Source Code">Source Code</a>
+ </li>
+ <li class="none">
+ <a href="team-list.html" title="Project Team">Project Team</a>
+ </li>
+ <li class="none">
+ <a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a>
+ </li>
+ <li class="none">
+ <a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a>
+ </li>
+ </ul>
+ <h5>ASF</h5>
+ <ul>
+ <li class="none">
+ <a href="http://www.apache.org/foundation/how-it-works.html" class="externalLink" title="How Apache Works">How Apache Works</a>
+ </li>
+ <li class="none">
+ <a href="http://www.apache.org/foundation/" class="externalLink" title="Foundation">Foundation</a>
+ </li>
+ <li class="none">
+ <a href="http://www.apache.org/foundation/sponsorship.html" class="externalLink" title="Sponsoring Apache">Sponsoring Apache</a>
+ </li>
+ <li class="none">
+ <a href="http://www.apache.org/foundation/thanks.html" class="externalLink" title="Thanks">Thanks</a>
+ </li>
+ </ul>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="poweredBy" alt="Built by Maven" src="./images/logos/maven-feather.png" />
+ </a>
+
+
+ </div>
+ </div>
+ <div id="bodyColumn">
+ <div id="contentBox">
+ <!-- -
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License. --><p></p><div class="section"><h2>Getting Started<a name="Getting_Started"></a></h2><p>This guide describes the steps required to install, deploy and validate the Apache Knox Gateway.</p><p></p></div><div class="section"><h2>Requirements<a name="Requirements"></a></h2><p>The following prerequisites must be installed to successfully complete the steps described in this guide.</p><div class="section"><h3>Java<a name="Java"></a></h3><p>Java 1.6 or later</p></div><div class="section"><h3>Hadoop<a name="Hadoop"></a></h3><p>A local installation of a Hadoop Cluster is required at this time. Hadoop EC2 cluster and/or Sandbox installations are currently difficult to access remotely via the Gateway. The EC2 and Sandbox limitation is caused by Hadoop services running with internal IP addresses. For the Gateway to work in these cases it will need to be deployed on the EC2 cluster or Sandbox, at this time.</p><p>The instructions that follow assume that the
Gateway is <i>not</i> collocated with the Hadoop clusters themselves and (most importantly) that the hostnames and IP addresses of the cluster services are accessible by the gateway where ever it happens to be running.</p><p>The Hadoop cluster should be ensured to have WebHDFS, WebHCat (i.e. Templeton) and Oozie configured, deployed and running.</p><p></p></div></div><div class="section"><h2>Installation<a name="Installation"></a></h2><div class="section"><h3>1. Extract the distribution ZIP<a name="a1._Extract_the_distribution_ZIP"></a></h3><p>Download and extract the gateway-0.2.0-SNAPSHOT.zip file into the installation directory that will contain your <tt>{GATEWAY_HOME}</tt></p>
+<div class="source"><pre>jar xf gateway-0.2.0-SNAPSHOT.zip
+</pre></div><p>This will create a directory <tt>gateway-0.2.0-SNAPSHOT</tt> in your current directory.</p></div><div class="section"><h3>2. Enter the <tt>{GATEWAY_HOME}</tt> directory<a name="a2._Enter_the_GATEWAY_HOME_directory"></a></h3>
+<div class="source"><pre>cd gateway-0.2.0-SNAPSHOT
+</pre></div><p>The fully qualified name of this directory will be referenced as <tt>{GATEWAY_HOME}</tt> throughout the remainder of this document.</p></div><div class="section"><h3>3. Start the demo LDAP server (ApacheDS)<a name="a3._Start_the_demo_LDAP_server_ApacheDS"></a></h3><p>First, understand that the LDAP server provided here is for demonstration purposes. You may configure the LDAP specifics within the topology descriptor for the cluster as described in step 5 below, in order to customize what LDAP instance to use. The assumption is that most users will leverage the demo LDAP server while evaluating this release and should therefore continue with the instructions here in step 3.</p><p>Edit <tt>{GATEWAY_HOME}/conf/users.ldif</tt> if required and add your users and groups to the file. A number of normal Hadoop users (e.g. hdfs, mapred, hcat, hive) have already been included. Note that the passwords in this file are fictitious and have nothing to do with the actual acc
ounts on the Hadoop cluster you are using. There is also a copy of this file in the templates directory that you can use to start over if necessary.</p><p>Start the LDAP server - pointing it to the config dir where it will find the users.ldif file in the conf directory.</p>
+<div class="source"><pre>java -jar bin/ldap-0.2.0-SNAPSHOT.jar conf &
+</pre></div><p>There are a number of log messages of the form <tt>Created null.</tt> that can safely be ignored. Take note of the port on which it was started as this needs to match later configuration. This will create a directory named org.apache.hadoop.gateway.security.EmbeddedApacheDirectoryServer that can safely be ignored.</p></div><div class="section"><h3>4. Start the Gateway server<a name="a4._Start_the_Gateway_server"></a></h3>
+<div class="source"><pre>java -jar bin/gateway-server-0.2.0-SNAPSHOT.jar
+</pre></div><p>Take note of the port identified in the logging output as you will need this for accessing the gateway.</p><p>The server will prompt you for the master secret (password). This secret is used to secure artifacts used to secure artifacts used by the gateway server for things like SSL, credential/password aliasing. This secret will have to be entered at startup unless you choose to persist it. Remember this secret and keep it safe. It represents the keys to the kingdom. See the Persisting the Master section for more information.</p></div><div class="section"><h3>5. Configure the Gateway with the topology of your Hadoop cluster<a name="a5._Configure_the_Gateway_with_the_topology_of_your_Hadoop_cluster"></a></h3><p>Edit the file <tt>{GATEWAY_HOME}/deployments/sample.xml</tt></p><p>Change the host and port in the urls of the <tt><service></tt> elements for NAMENODE, TEMPLETON and OOZIE services to match your Hadoop cluster deployment.</p><p>The default configu
ration contains the LDAP URL for a LDAP server. By default that file is configured to access the demo ApacheDS based LDAP server and its default configuration. By default, this server listens on port 33389. Optionally, you can change the LDAP URL for the LDAP server to be used for authentication. This is set via the main.ldapRealm.contextFactory.url property in the <tt><gateway><provider><authentication></tt> section.</p><p>Save the file. The directory {GATEWAY_HOME}/deployments is monitored by the Gateway server and reacts to the discovery of a new or changed cluster topology descriptor by provisioning the endpoints and required filter chains to serve the needs of each cluster as described by the topology file. Note that the name of the file excluding the extension is also used as the path for that cluster in the URL. So for example the sample.xml file will result in Gateway URLs of the form <tt>http://{gateway-host}:{gateway-port}/gateway/sample/namenode/
api/v1</tt></p></div><div class="section"><h3>6. Test the installation and configuration of your Gateway<a name="a6._Test_the_installation_and_configuration_of_your_Gateway"></a></h3><p>Invoke the LISTSATUS operation on HDFS represented by your configured NAMENODE by using your web browser or curl:</p>
+<div class="source"><pre>curl -i -k -u hdfs:hdfs-password -X GET \
+'https://localhost:8443/gateway/sample/namenode/api/v1/?op=LISTSTATUS'
+</pre></div><p>The results of the above command should result in something to along the lines of the output below. The exact information returned is subject to the content within HDFS in your Hadoop cluster.</p>
+<div class="source"><pre>HTTP/1.1 200 OK
+Content-Type: application/json
+Content-Length: 760
+Server: Jetty(6.1.26)
+
+{"FileStatuses":{"FileStatus":[
+{"accessTime":0,"blockSize":0,"group":"hdfs","length":0,"modificationTime":1350595859762,"owner":"hdfs","pathSuffix":"apps","permission":"755","replication":0,"type":"DIRECTORY"},
+{"accessTime":0,"blockSize":0,"group":"mapred","length":0,"modificationTime":1350595874024,"owner":"mapred","pathSuffix":"mapred","permission":"755","replication":0,"type":"DIRECTORY"},
+{"accessTime":0,"blockSize":0,"group":"hdfs","length":0,"modificationTime":1350596040075,"owner":"hdfs","pathSuffix":"tmp","permission":"777","replication":0,"type":"DIRECTORY"},
+{"accessTime":0,"blockSize":0,"group":"hdfs","length":0,"modificationTime":1350595857178,"owner":"hdfs","pathSuffix":"user","permission":"755","replication":0,"type":"DIRECTORY"}
+]}}
+</pre></div><p>For additional information on WebHDFS, Templeton/WebHCat and Oozie REST APIs, see the following URLs respectively:</p>
+<ul>
+ <li>WebHDFS - <a class="externalLink" href="http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html">http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html</a></li>
+ <li>Templeton/WebHCat - <a class="externalLink" href="http://people.apache.org/~thejas/templeton_doc_v1/">http://people.apache.org/~thejas/templeton_doc_v1/</a></li>
+ <li>Oozie - <a class="externalLink" href="http://oozie.apache.org/docs/3.3.1/WebServicesAPI.html">http://oozie.apache.org/docs/3.3.1/WebServicesAPI.html</a></li>
+</ul><p></p></div></div><div class="section"><h2>Examples<a name="Examples"></a></h2><p>More examples can be found <a href="/examples.html">here</a>.</p><p></p></div><div class="section"><h2>Persisting the Master Secret<a name="Persisting_the_Master_Secret"></a></h2><p>The master secret is required to start the server. This secret is used to access secured artifacts by the gateway instance. Keystore, trust stores and credential stores are all protected with the master secret.</p><p>You may persist the master secret by supplying the <i>-persist-master</i> switch at startup. This will result in a warning indicating that persisting the secret is less secure than providing it at startup. We do make some provisions in order to protect the persisted password.</p><p>It is encrypted with AES 128 bit encryption and where possible the file permissions are set to only be accessable by the user that the gateway is running as.</p><p>After persisting the secret, ensure that the file at co
nfig/security/master has the appropriate permissions set for your environment. This is probably the most important layer of defense for master secret. Do not assume that the encryption if sufficient protection.</p><p>A specific user should be created to run the gateway this will protect a persisted master file.</p><p></p></div><div class="section"><h2>Management of Security Artifacts<a name="Management_of_Security_Artifacts"></a></h2><p>There are a number of artifacts that are used by the gateway in ensuring the security of wire level communications, access to protected resources and the encryption of sensitive data. These artifacts can be managed from outside of the gateway instances or generated and populated by the gateway instance itself.</p><p>The following is a description of how this is coordinated with both standalone (development, demo, etc) gateway instances and instances as part of a cluster of gateways in mind.</p><p>Upon start of the gateway server we:</p>
+<ol style="list-style-type: decimal">
+ <li><p>Look for an identity store at conf/security/keystores/gateway.jks. The identity store contains the certificate and private key used to represent the identity of the server for SSL connections and signature creation.</p>
+ <ul>
+ <li>If there is no identity store we create one and generate a self-signed certificate for use in standalone/demo mode. The certificate is stored with an alias of gateway-identity.</li>
+ <li>If there is an identity store found than we ensure that it can be loaded using the provided master secret and that there is an alias with called gateway-identity.</li>
+ </ul></li>
+ <li><p>Look for a credential store at <tt>conf/security/keystores/__gateway-credentials.jceks</tt>. This credential store is used to store secrets/passwords that are used by the gateway. For instance, this is where the pass-phrase for accessing the gateway-identity certificate is kept.</p></li>
+</ol>
+<ul>
+ <li>If there is no credential store found then we create one and populate it with a generated pass-phrase for the alias <tt>gateway-identity-passphrase</tt>. This is coordinated with the population of the self-signed cert into the identity-store.</li>
+ <li>If a credential store is found then we ensure that it can be loaded using the provided master secret and that the expected aliases have been populated with secrets.</li>
+</ul><p>Upon deployment of a Hadoop cluster topology within the gateway we:</p>
+<ol style="list-style-type: decimal">
+ <li><p>Look for a credential store for the topology. For instance, we have a sample topology that gets deployed out of the box. We look for <tt>conf/security/keystores/sample-credentials.jceks</tt>. This topology specific credential store is used for storing secrets/passwords that are used for encrypting sensitive data with topology specific keys.</p>
+ <ul>
+ <li>If no credential store is found for the topology being deployed then one is created for it. Population of the aliases is delegated to the configured providers within the system that will require the use of a secret for a particular task. They may programmatic set the value of the secret or choose to have the value for the specified alias generated through the AliasService.</li>
+ <li>If a credential store is found then we ensure that it can be loaded with the provided master secret and the configured providers have the opportunity to ensure that the aliases are populated and if not to populate them.</li>
+ </ul></li>
+</ol><p>By leveraging the algorithm described above we can provide a window of opportunity for management of these artifacts in a number of ways.</p>
+<ol style="list-style-type: decimal">
+ <li>Using a single gateway instance as a master instance the artifacts can be generated or placed into the expected location and then replicated across all of the slave instances before startup.</li>
+ <li>Using an NFS mount as a central location for the artifacts would provide a single source of truth without the need to replicate them over the network. Of course, NFS mounts have their own challenges.</li>
+</ol><p>Summary of Secrets to be Managed:</p>
+<ol style="list-style-type: decimal">
+ <li>Master secret - the same for all gateway instances in a cluster of gateways</li>
+ <li>All security related artifacts are protected with the master secret</li>
+ <li>Secrets used by the gateway itself are stored within the gateway credential store and are the same across all gateway instances in the cluster of gateways</li>
+ <li>Secrets used by providers within cluster topologies are stored in topology specific credential stores and are the same for the same topology across the cluster of gateway instances. However, they are specific to the topology - so secrets for one hadoop cluster are different from those of another. This allows for fail-over from one gateway instance to another even when encryption is being used while not allowing the compromise of one encryption key to expose the data for all clusters.</li>
+</ol><p>NOTE: the SSL certificate will need special consideration depending on the type of certificate. Wildcard certs may be able to be shared across all gateway instances in a cluster. When certs are dedicated to specific machines the gateway identity store will not be able to be blindly replicated as hostname verification problems will ensue. Obviously, trust-stores will need to be taken into account as well.</p><p></p></div><div class="section"><h2>Mapping Gateway URLs to Hadoop cluster URLs<a name="Mapping_Gateway_URLs_to_Hadoop_cluster_URLs"></a></h2><p>The Gateway functions much like a reverse proxy. As such it maintains a mapping of URLs that are exposed externally by the Gateway to URLs that are provided by the Hadoop cluster. Examples of mappings for the NameNode and Templeton are shown below. These mapping are generated from the combination of the Gateway configuration file (i.e. {GATEWAY_HOME}/gateway-site.xml) and the cluster topology descriptors (e.g. {GATEWAY_
HOME}/deployments/{cluster-name}.xml).</p>
+<ul>
+ <li>HDFS (NameNode)
+ <ul>
+ <li>Gateway: <tt>http://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/namenode/api/v1</tt></li>
+ <li>Cluster: <tt>http://{namenode-host}:50070/webhdfs/v1</tt></li>
+ </ul></li>
+ <li>WebHCat (Templeton)
+ <ul>
+ <li>Gateway: <tt>http://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/templeton/api/v1</tt></li>
+ <li>Cluster: <tt>http://{templeton-host}:50111/templeton/v1</tt></li>
+ </ul></li>
+ <li>Oozie
+ <ul>
+ <li>Gateway: <tt>http://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/oozie/api/v1</tt></li>
+ <li>Cluster: <tt>http://{templeton-host}:11000/oozie/v1</tt></li>
+ </ul></li>
+</ul><p>The values for <tt>{gateway-host}</tt>, <tt>{gateway-port}</tt>, <tt>{gateway-path}</tt> are provided via the Gateway configuration file (i.e. <tt>{GATEWAY_HOME}/gateway-site.xml</tt>).</p><p>The value for <tt>{cluster-name}</tt> is derived from the name of the cluster topology descriptor (e.g. <tt>{GATEWAY_HOME}/deployments/{cluster-name}.xml</tt>).</p><p>The value for <tt>{namenode-host}</tt> and <tt>{templeton-host}</tt> is provided via the cluster topology descriptor (e.g. <tt>{GATEWAY_HOME}/deployments/{cluster-name}.xml</tt>).</p><p>Note: The ports 50070, 50111 and 11000 are the defaults for NameNode, Templeton and Oozie respectively. Their values can also be provided via the cluster topology descriptor if your Hadoop cluster uses different ports.</p><p></p></div><div class="section"><h2>Enabling logging<a name="Enabling_logging"></a></h2><p>If necessary you can enable additional logging by editing the <tt>log4j.properties</tt> file in the <tt>conf</tt> dire
ctory. Changing the rootLogger value from <tt>ERROR</tt> to <tt>DEBUG</tt> will generate a large amount of debug logging. A number of useful, more fine loggers are also provided in the file.</p><p></p></div><div class="section"><h2>Filing bugs<a name="Filing_bugs"></a></h2><p>File bugs at hortonworks.jira.com under Project Hadoop Gateway Development Include the results of</p>
+<div class="source"><pre>java -jar bin/gateway-0.2.0-SNAPSHOT.jar -version
+</pre></div><p>in the Environment section. Also include the version of Hadoop being used.</p><p></p></div><div class="section"><h2>Disclaimer<a name="Disclaimer"></a></h2><p>The Apache Knox Gateway is an effort undergoing incubation at the Apache Software Foundation (ASF), sponsored by the Apache Incubator PMC.</p><p>Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects.</p><p>While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.</p></div>
+ </div>
+ </div>
+ <div class="clear">
+ <hr/>
+ </div>
+ <div id="footer">
+ <div class="xright">
+ <?xml version="1.0" encoding="UTF-8"?>
+<footer>
+ <div class="row span12">Apache Knox Gateway, Apache, the Apache feather logo and the Apache Knox Gateway project logos
+ are trademarks of The Apache Software Foundation.
+ All other marks mentioned may be trademarks or registered trademarks of their respective owners.</div>
+ <div class="row span12">
+ <a href="privacy-policy.html">Privacy Policy</a>
+ </div>
+</footer>
+ </div>
+ <div class="clear">
+ <hr/>
+ </div>
+ </div>
+ </body>
+</html>
Added: incubator/knox/site/release-0-2-0.html
URL: http://svn.apache.org/viewvc/incubator/knox/site/release-0-2-0.html?rev=1458909&view=auto
==============================================================================
--- incubator/knox/site/release-0-2-0.html (added)
+++ incubator/knox/site/release-0-2-0.html Wed Mar 20 16:06:11 2013
@@ -0,0 +1,185 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<!-- Generated by Apache Maven Doxia at Mar 20, 2013 -->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
+ <title></title>
+ <style type="text/css" media="all">
+ @import url("./css/maven-base.css");
+ @import url("./css/maven-theme.css");
+ @import url("./css/site.css");
+ </style>
+ <link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
+ <meta name="Date-Revision-yyyymmdd" content="20130320" />
+ <meta http-equiv="Content-Language" content="en" />
+
+ </head>
+ <body class="composite">
+ <div id="banner">
+ <a href="./" id="bannerLeft">
+ <img src="images/apache-logo.gif" alt="Knox Gateway" />
+ </a>
+ <a href=".." id="bannerRight">
+ <img src="images/apache-incubator-logo.png" alt="Apache Incubator" />
+ </a>
+ <div class="clear">
+ <hr/>
+ </div>
+ </div>
+ <div id="breadcrumbs">
+
+
+ <div class="xleft">
+ <a href="index.html" title="Knox">Knox</a>
+ >
+
+ </div>
+ <div class="xright"> <a href="https://git-wip-us.apache.org/repos/asf/incubator-knox.git" class="externalLink" title="Git">Git</a>
+ |
+ <a href="https://svn.apache.org/repos/asf/incubator/knox/site" class="externalLink" title="Svn">Svn</a>
+ |
+ <a href="https://issues.apache.org/jira/browse/KNOX" class="externalLink" title="Jira">Jira</a>
+ |
+ <a href="https://cwiki.apache.org/confluence/display/KNOX/Knox" class="externalLink" title="Wiki">Wiki</a>
+
+
+ | <span id="publishDate">Last Published: 2013-03-20</span>
+ | <span id="projectVersion">Version: 0.2.0-SNAPSHOT</span>
+ </div>
+ <div class="clear">
+ <hr/>
+ </div>
+ </div>
+ <div id="leftColumn">
+ <div id="navcolumn">
+
+
+ <h5>Knox</h5>
+ <ul>
+ <li class="none">
+ <a href="index.html" title="Welcome">Welcome</a>
+ </li>
+ <li class="none">
+ <a href="news.html" title="News">News</a>
+ </li>
+ <li class="none">
+ <a href="license.html" title="License">License</a>
+ </li>
+ </ul>
+ <h5>Documentation</h5>
+ <ul>
+ <li class="none">
+ <a href="getting-started.html" title="Getting Started">Getting Started</a>
+ </li>
+ <li class="none">
+ <a href="examples.html" title="Usage Examples">Usage Examples</a>
+ </li>
+ <li class="none">
+ <a href="client.html" title="Client (KnoxShell DSL)">Client (KnoxShell DSL)</a>
+ </li>
+ <li class="none">
+ <a href="https://cwiki.apache.org/confluence/display/KNOX/Knox" class="externalLink" title="Wiki">Wiki</a>
+ </li>
+ </ul>
+ <h5>Releases</h5>
+ <ul>
+ <li class="none">
+ <strong>0.2.0 (coming soon)</strong>
+ </li>
+ <li class="none">
+ <a href="roadmap-0-3-0.html" title="0.3.0 (planning)">0.3.0 (planning)</a>
+ </li>
+ </ul>
+ <h5>Processes</h5>
+ <ul>
+ <li class="none">
+ <a href="build-process.html" title="Build">Build</a>
+ </li>
+ <li class="none">
+ <a href="release-process.html" title="Release">Release</a>
+ </li>
+ <li class="none">
+ <a href="contribute-process.html" title="Contribute">Contribute</a>
+ </li>
+ <li class="none">
+ <a href="site-process.html" title="Site Publication">Site Publication</a>
+ </li>
+ </ul>
+ <h5>Resources</h5>
+ <ul>
+ <li class="none">
+ <a href="https://git-wip-us.apache.org/repos/asf/incubator-knox.git" class="externalLink" title="Source Code">Source Code</a>
+ </li>
+ <li class="none">
+ <a href="team-list.html" title="Project Team">Project Team</a>
+ </li>
+ <li class="none">
+ <a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a>
+ </li>
+ <li class="none">
+ <a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a>
+ </li>
+ </ul>
+ <h5>ASF</h5>
+ <ul>
+ <li class="none">
+ <a href="http://www.apache.org/foundation/how-it-works.html" class="externalLink" title="How Apache Works">How Apache Works</a>
+ </li>
+ <li class="none">
+ <a href="http://www.apache.org/foundation/" class="externalLink" title="Foundation">Foundation</a>
+ </li>
+ <li class="none">
+ <a href="http://www.apache.org/foundation/sponsorship.html" class="externalLink" title="Sponsoring Apache">Sponsoring Apache</a>
+ </li>
+ <li class="none">
+ <a href="http://www.apache.org/foundation/thanks.html" class="externalLink" title="Thanks">Thanks</a>
+ </li>
+ </ul>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="poweredBy" alt="Built by Maven" src="./images/logos/maven-feather.png" />
+ </a>
+
+
+ </div>
+ </div>
+ <div id="bodyColumn">
+ <div id="contentBox">
+ <!-- -
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License. --><div class="section"><h2>Introduction<a name="Introduction"></a></h2><p>TODO</p></div><div class="section"><h2>Disclaimer<a name="Disclaimer"></a></h2><p>The Apache Knox Gateway is an effort undergoing incubation at the Apache Software Foundation (ASF), sponsored by the Apache Incubator PMC.</p><p>Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects.</p><p>While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.</p></div>
+ </div>
+ </div>
+ <div class="clear">
+ <hr/>
+ </div>
+ <div id="footer">
+ <div class="xright">
+ <?xml version="1.0" encoding="UTF-8"?>
+<footer>
+ <div class="row span12">Apache Knox Gateway, Apache, the Apache feather logo and the Apache Knox Gateway project logos
+ are trademarks of The Apache Software Foundation.
+ All other marks mentioned may be trademarks or registered trademarks of their respective owners.</div>
+ <div class="row span12">
+ <a href="privacy-policy.html">Privacy Policy</a>
+ </div>
+</footer>
+ </div>
+ <div class="clear">
+ <hr/>
+ </div>
+ </div>
+ </body>
+</html>