You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@knox.apache.org by km...@apache.org on 2013/09/26 20:21:05 UTC
svn commit: r1526638 [3/3] - in /incubator/knox:
site/books/knox-incubating-0-3-0/ trunk/books/0.3.0/ trunk/books/common/
trunk/markbook/src/main/java/org/apache/hadoop/gateway/markbook/
Modified: incubator/knox/trunk/books/0.3.0/book.md
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/0.3.0/book.md?rev=1526638&r1=1526637&r2=1526638&view=diff
==============================================================================
--- incubator/knox/trunk/books/0.3.0/book.md (original)
+++ incubator/knox/trunk/books/0.3.0/book.md Thu Sep 26 18:21:04 2013
@@ -24,24 +24,24 @@
-Apache Knox Gateway 0.3.0 (Incubator)
-=====================================
+# Apache Knox Gateway 0.3.0 (Incubator) #
-
-Table Of Contents
------------------
+## Table Of Contents ##
* [Introduction](#Introduction)
-* [Download](#Download)
-* [Installation](#Installation)
* [Getting Started](#Getting+Started)
-* [Supported Services](#Supported+Services)
-* [Sandbox Configuration](#Sandbox+Configuration)
-* [Usage Examples](#Usage+Examples)
+ * [Requirements](#Requirements)
+ * [Download](#Download)
+ * [Verify](#Verify)
+ * [Install](#Install)
+ * [Supported Services](#Supported+Services)
+ * [Basic Usage](#Basic+Usage)
+ * [Sandbox Configuration](#Sandbox+Configuration)
* [Gateway Details](#Gateway+Details)
* [Authentication](#Authentication)
* [Authorization](#Authorization)
* [Configuration](#Configuration)
+ * [Secure Clusters](#Secure+Clusters)
* [Client Details](#Client+Details)
* [Service Details](#Service+Details)
* [WebHDFS](#WebHDFS)
@@ -49,161 +49,31 @@ Table Of Contents
* [Oozie](#Oozie)
* [HBase/Starbase](#HBase)
* [Hive](#Hive)
-* [Secure Clusters](#Secure+Clusters)
* [Trouble Shooting](#Trouble+Shooting)
-* [Release Verification](#Release+Verification)
* [Export Controls](#Export+Controls)
-{{Introduction}}
-------------------------------
-
-TODO
-
-
-{{Requirements}}
-----------------
-
-### Java ###
-
-Java 1.6 or later is required for the Knox Gateway runtime.
-Use the command below to check the version of Java installed on the system where Knox will be running.
-
- java -version
-
-### Hadoop ###
-
-An an existing Hadoop 1.x or 2.x cluster is required for Knox to protect.
-One of the easiest ways to ensure this it to utilize a HDP Sandbox VM.
-It is possible to use a Hadoop cluster deployed on EC2 but this will require additional configuration.
-Currently if this Hadoop cluster is secured with Kerberos only WebHDFS will work and additional configuration is required.
-
-The Hadoop cluster should be ensured to have at least WebHDFS, WebHCat (i.e. Templeton) and Oozie configured, deployed and running.
-HBase/Stargate and Hive can also be accessed via the Knox Gateway given the proper versions and configuration.
-
-The instructions that follow assume that the Gateway is *not* collocated with the Hadoop clusters themselves and (most importantly) that the hostnames and IP addresses of the cluster services are accessible by the gateway where ever it happens to be running.
-All of the instructions and samples are tailored to work "out of the box" against a Hortonworks Sandbox 2.x VM.
-
-This release of the Apache Knox Gateway has been tested against the [Hortonworks Sandbox 2.0](http://hortonworks.com/products/hortonworks-sandbox/).
-
-
-{{Download}}
-------------
-
-Download and extract the knox-\{VERSION\}.zip}} file into the installation directory that will contain your {{\{GATEWAY_HOME\}}}.
-You can find the downloads for Knox releases on the [Apache mirrors|http://www.apache.org/dyn/closer.cgi/incubator/knox/].
-
-* Source archive: [knox-incubating-0.3.0-src.zip][src-zip] ([PGP signature][src-pgp], [SHA1 digest][src-sha], [MD5 digest][src-md5])
-* Binary archive: [knox-incubating-0.3.0.zip][bin-zip] ([PGP signature][bin-pgp], [SHA1 digest][bin-sha], [MD5 digest][bin-md5])
-
-[src-zip]: http://www.apache.org/dyn/closer.cgi/incubator/knox/0.3.0/knox-incubating-0.3.0-src.zip
-[src-sha]: http://www.apache.org/dist/incubator/knox/0.3.0/knox-incubating-0.3.0-src.zip.sha
-[src-pgp]: http://www.apache.org/dist/incubator/knox/0.3.0/knox-0.3.0-incubating-src.zip.asc
-[src-md5]: http://www.apache.org/dist/incubator/knox/0.3.0/knox-incubating-0.3.0-src.zip.md5
-[bin-zip]: http://www.apache.org/dyn/closer.cgi/incubator/knox/0.3.0/knox-incubating-0.3.0.zip
-[bin-pgp]: http://www.apache.org/dist/incubator/knox/0.3.0/knox-incubating-0.3.0.zip.asc
-[bin-sha]: http://www.apache.org/dist/incubator/knox/0.3.0/knox-incubating-0.3.0.zip.sha
-[bin-md5]: http://www.apache.org/dist/incubator/knox/0.3.0/knox-incubating-0.3.0.zip.md5
-
-| ![$] Important |
-| -------------- |
-| Please ensure that you validate the integrity of any downloaded files as described [below](#Release+Verification). |
-
-Apache Knox Gateway releases are available under the [Apache License, Version 2.0][asl].
-See the NOTICE file contained in each release artifact for applicable copyright attribution notices.
-
-
-<<install.md>>
-<<using.md>>
-
-
-{{Supported Services}}
-----------------------
-
-This table enumerates the versions of various Hadoop services that have been tested to work with the Knox Gateway.
-Only more recent versions of some Hadoop components when secured via Kerberos can be accessed via the Knox Gateway.
-
-| Service | Version | Non-Secure | Secure |
-| ----------------- | ---------- | ----------- | ------ |
-| WebHDFS | 2.1.0 | ![y] | ![?]![y] |
-| WebHCat/Templeton | 0.11.0 | ![y] | ![?]![n] |
-| Ozzie | 4.0.0 | ![y] | ![?] |
-| HBase/Stargate | 0.95.2 | ![y] | ![?] |
-| Hive/JDBC | 0.11.0 | ![n] | ![n] |
-| | 0.12.0 | ![?]![y] | ![?] |
-| Hive/ODBC | 0.12.0 | ![?] | ![?] |
-
-ProxyUser feature of WebHDFS, WebHCat and Oozie required for secure cluster support seem to work fine.
-Knox code seems to be broken for support of secure cluster at this time for WebHDFS, WebHCat and Oozie.
-
-
-<<sandbox.md>>
-
-
-{{Usage Examples}}
-------------------
-
-These examples provide more detail about how to access various Apache Hadoop services via the Apache Knox Gateway.
-
-* [WebHDFS](#WebHDFS+Examples)
-* [WebHCat/Templeton](#WebHCat+Examples)
-* [Oozie](#Oozie+Examples)
-* [HBase](#HBase+Examples)
-* [Hive](#Hive+Examples)
-
-<<config.md>>
-
-{{Gateway Details}}
--------------------
-
-TODO
-
-<<config.md>>
-<<authn.md>>
-<<authz.md>>
-<<client.md>>
-
-{{Service Details}}
--------------------
-
-TODO
-
-<<webhdfs.md>>
-<<webhcat.md>>
-<<oozie.md>>
-<<hbase.md>>
-<<hive.md>>
-<<kerberos.md>>
-<<trouble.md>>
-
-
-{{Release Verification}}
-------------------------
-
-It is essential that you verify the integrity of the downloaded files using the PGP signatures.
-Please read Verifying Apache HTTP Server Releases for more information on why you should verify our releases.
-
-The PGP signatures can be verified using PGP or GPG.
-First download the KEYS file as well as the .asc signature files for the relevant release packages.
-Make sure you get these files from the main distribution directory, rather than from a mirror.
-Then verify the signatures using one of the methods below.
-
- % pgpk -a KEYS
- % pgpv knox-incubating-0.3.0.zip.asc
+## Introduction ##
-or
+The Apache Knox Gateway is a system that provides a single point of authentication and access for Apache Hadoop services in a cluster.
+The goal is to simplify Hadoop security for both users (i.e. who access the cluster data and execute jobs) and operators (i.e. who control access and manage the cluster).
+The gateway runs as a server (or cluster of servers) that provide centralized access to one or more Hadoop clusters.
+In general the goals of the gateway are as follows:
- % pgp -ka KEYS
- % pgp knox-incubating-0.3.0.zip.asc
+* Provide perimeter security for Hadoop REST APIs to make Hadoop security setup easier
+* Support authentication and token verification security scenarios
+* Deliver users a single URL end-point that aggregates capabilities for data and jobs
+* Enable integration with enterprise and cloud identity management environments
-or
- % gpg --import KEYS
- % gpg --verify knox-incubating-0.3.0.zip.asc
+<<book_getting-started.md>>
+<<book_gateway-details.md>>
+<<book_client-details.md>>
+<<book_service-details.md>>
+<<book_trouble-shooting.md>>
-{{Export Controls}}
--------------------
+## Export Controls ##
Apache Knox Gateway includes cryptographic software.
The country in which you currently reside may have restrictions on the import, possession, use, and/or re-export to another country, of encryption software.
Added: incubator/knox/trunk/books/0.3.0/book_gateway-details.md
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/0.3.0/book_gateway-details.md?rev=1526638&view=auto
==============================================================================
--- incubator/knox/trunk/books/0.3.0/book_gateway-details.md (added)
+++ incubator/knox/trunk/books/0.3.0/book_gateway-details.md Thu Sep 26 18:21:04 2013
@@ -0,0 +1,56 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+--->
+
+{{Gateway Details}}
+-------------------
+
+TODO
+
+### Mapping Gateway URLs to Hadoop cluster URLs
+
+The Gateway functions much like a reverse proxy.
+As such it maintains a mapping of URLs that are exposed externally by the gateway to URLs that are provided by the Hadoop cluster.
+Examples of mappings for the WebHDFS, WebHCat, Oozie and Stargate/Hive are shown below.
+These mapping are generated from the combination of the gateway configuration file (i.e. `{GATEWAY_HOME}/conf/gateway-site.xml`) and the cluster topology descriptors (e.g. `{GATEWAY_HOME}/deployments/{cluster-name}.xml`).
+
+* WebHDFS
+ * Gateway: `https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/webhdfs`
+ * Cluster: `http://{webhdfs-host}:50070/webhdfs`
+* WebHCat (Templeton)
+ * Gateway: `https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/templeton`
+ * Cluster: `http://{webhcat-host}:50111/templeton}`
+* Oozie
+ * Gateway: `https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/oozie`
+ * Cluster: `http://{oozie-host}:11000/oozie}`
+* Stargate (HBase)
+ * Gateway: `https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/hbase`
+ * Cluster: `http://{hbase-host}:60080`
+
+The values for `{gateway-host}`, `{gateway-port}`, `{gateway-path}` are provided via the Gateway configuration file (i.e. `{GATEWAY_HOME}/conf/gateway-site.xml`).
+
+The value for `{cluster-name}` is derived from the name of the cluster topology descriptor (e.g. `{GATEWAY_HOME}/deployments/{cluster-name}.xml`).
+
+The value for `{webhdfs-host}` and `{webhcat-host}` are provided via the cluster topology descriptor (e.g. `{GATEWAY_HOME}/deployments/{cluster-name}.xml`).
+
+Note: The ports 50070, 50111, 11000 and 60080 are the defaults for WebHDFS, WebHCat, Oozie and Stargate/HBase respectively.
+Their values can also be provided via the cluster topology descriptor if your Hadoop cluster uses different ports.
+
+<<config.md>>
+<<config_authn.md>>
+<<config_authz.md>>
+<<config_kerberos.md>>
+
Copied: incubator/knox/trunk/books/0.3.0/book_getting-started.md (from r1526490, incubator/knox/trunk/books/0.3.0/using.md)
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/0.3.0/book_getting-started.md?p2=incubator/knox/trunk/books/0.3.0/book_getting-started.md&p1=incubator/knox/trunk/books/0.3.0/using.md&r1=1526490&r2=1526638&rev=1526638&view=diff
==============================================================================
--- incubator/knox/trunk/books/0.3.0/using.md (original)
+++ incubator/knox/trunk/books/0.3.0/book_getting-started.md Thu Sep 26 18:21:04 2013
@@ -15,22 +15,144 @@
limitations under the License.
--->
-{{Getting Started}}
--------------------
+## Getting Started ##
-### 2. Enter the `{GATEWAY_HOME}` directory
+This section provides everything you need to know to get the gateway up and running against a Sandbox VM Hadoop cluster.
+
+
+### Requirements ###
+
+#### Java ####
+
+Java 1.6 or later is required for the Knox Gateway runtime.
+Use the command below to check the version of Java installed on the system where Knox will be running.
+
+ java -version
+
+#### Hadoop ####
+
+An an existing Hadoop 1.x or 2.x cluster is required for Knox to protect.
+One of the easiest ways to ensure this it to utilize a Hortonworks Sandbox VM.
+It is possible to use a Hadoop cluster deployed on EC2 but this will require additional configuration not covered here.
+It is also possible to use a limited set of services in Hadoop cluster secured with Kerberos.
+This too required additional configuration that is not described here.
+
+The Hadoop cluster should be ensured to have at least WebHDFS, WebHCat (i.e. Templeton) and Oozie configured, deployed and running.
+HBase/Stargate and Hive can also be accessed via the Knox Gateway given the proper versions and configuration.
+
+The instructions that follow assume a few things:
+
+1. The gateway is *not* collocated with the Hadoop clusters themselves
+2. The host names and IP addresses of the cluster services are accessible by the gateway where ever it happens to be running.
+
+All of the instructions and samples provided here are tailored and tested to work "out of the box" against a [Hortonworks Sandbox 2.x VM][sandbox].
+
+
+### Download ###
+
+Download and extract the knox-{VERSION}.zip file into the installation directory.
+This directory will be referred to as your `{GATEWAY_HOME}`.
+You can find the downloads for Knox releases on the [Apache mirrors][mirror].
+
+* Source archive: [knox-incubating-0.3.0-src.zip][src-zip] ([PGP signature][src-pgp], [SHA1 digest][src-sha], [MD5 digest][src-md5])
+* Binary archive: [knox-incubating-0.3.0.zip][bin-zip] ([PGP signature][bin-pgp], [SHA1 digest][bin-sha], [MD5 digest][bin-md5])
+
+[src-zip]: http://www.apache.org/dyn/closer.cgi/incubator/knox/0.3.0/knox-incubating-0.3.0-src.zip
+[src-sha]: http://www.apache.org/dist/incubator/knox/0.3.0/knox-incubating-0.3.0-src.zip.sha
+[src-pgp]: http://www.apache.org/dist/incubator/knox/0.3.0/knox-0.3.0-incubating-src.zip.asc
+[src-md5]: http://www.apache.org/dist/incubator/knox/0.3.0/knox-incubating-0.3.0-src.zip.md5
+[bin-zip]: http://www.apache.org/dyn/closer.cgi/incubator/knox/0.3.0/knox-incubating-0.3.0.zip
+[bin-pgp]: http://www.apache.org/dist/incubator/knox/0.3.0/knox-incubating-0.3.0.zip.asc
+[bin-sha]: http://www.apache.org/dist/incubator/knox/0.3.0/knox-incubating-0.3.0.zip.sha
+[bin-md5]: http://www.apache.org/dist/incubator/knox/0.3.0/knox-incubating-0.3.0.zip.md5
+
+Apache Knox Gateway releases are available under the [Apache License, Version 2.0][asl].
+See the NOTICE file contained in each release artifact for applicable copyright attribution notices.
+
+
+{{Verify}}
+------------------------
+
+It is essential that you verify the integrity of the downloaded files using the PGP signatures.
+Please read Verifying Apache HTTP Server Releases for more information on why you should verify our releases.
+
+The PGP signatures can be verified using PGP or GPG.
+First download the KEYS file as well as the .asc signature files for the relevant release packages.
+Make sure you get these files from the main distribution directory, rather than from a mirror.
+Then verify the signatures using one of the methods below.
+
+ % pgpk -a KEYS
+ % pgpv knox-incubating-0.3.0.zip.asc
+
+or
+
+ % pgp -ka KEYS
+ % pgp knox-incubating-0.3.0.zip.asc
+
+or
+
+ % gpg --import KEYS
+ % gpg --verify knox-incubating-0.3.0.zip.asc
+
+
+### Install ###
+
+#### ZIP ####
+
+Download and extract the `knox-{VERSION}.zip` file into the installation directory that will contain your `{GATEWAY_HOME}`.
+You can find the downloads for Knox releases on the [Apache mirrors][mirror].
+
+ jar xf knox-{VERSION}.zip
+
+This will create a directory `knox-{VERSION}` in your current directory.
+
+
+#### RPM ####
+
+TODO
+
+
+#### Layout ####
+
+TODO - Describe the purpose of all of the directories
+
+
+### Supported Services ###
+
+This table enumerates the versions of various Hadoop services that have been tested to work with the Knox Gateway.
+Only more recent versions of some Hadoop components when secured via Kerberos can be accessed via the Knox Gateway.
+
+| Service | Version | Non-Secure | Secure |
+| ----------------- | ---------- | ----------- | ------ |
+| WebHDFS | 2.1.0 | ![y] | ![?]![y] |
+| WebHCat/Templeton | 0.11.0 | ![y] | ![?]![n] |
+| Ozzie | 4.0.0 | ![y] | ![?] |
+| HBase/Stargate | 0.95.2 | ![y] | ![?] |
+| Hive/JDBC | 0.11.0 | ![n] | ![n] |
+| | 0.12.0 | ![?]![y] | ![?] |
+| Hive/ODBC | 0.12.0 | ![?] | ![?] |
+
+ProxyUser feature of WebHDFS, WebHCat and Oozie required for secure cluster support seem to work fine.
+Knox code seems to be broken for support of secure cluster at this time for WebHDFS, WebHCat and Oozie.
+
+
+### Basic Usage ###
+
+#### Starting Servers ####
+
+##### 1. Enter the `{GATEWAY_HOME}` directory
cd knox-{VERSION}
-The fully qualified name of this directory will be referenced as {{\{GATEWAY_HOME\}}} throughout the remainder of this document.
+The fully qualified name of this directory will be referenced as `{GATEWAY_HOME}}} throughout the remainder of this document.
-### 3. Start the demo LDAP server (ApacheDS)
+##### 2. Start the demo LDAP server (ApacheDS)
First, understand that the LDAP server provided here is for demonstration purposes.
You may configure the LDAP specifics within the topology descriptor for the cluster as described in step 5 below, in order to customize what LDAP instance to use.
The assumption is that most users will leverage the demo LDAP server while evaluating this release and should therefore continue with the instructions here in step 3.
-Edit {{\{GATEWAY_HOME\}/conf/users.ldif}} if required and add your users and groups to the file.
+Edit `{GATEWAY_HOME}/conf/users.ldif` if required and add your users and groups to the file.
A sample end user "bob" has been already included.
Note that the passwords in this file are "fictitious" and have nothing to do with the actual accounts on the Hadoop cluster you are using.
There is also a copy of this file in the templates directory that you can use to start over if necessary.
@@ -39,35 +161,45 @@ Start the LDAP server - pointing it to t
java -jar bin/ldap.jar conf &
-There are a number of log messages of the form {{Created null.}} that can safely be ignored. Take note of the port on which it was started as this needs to match later configuration.
+There are a number of log messages of the form {{Created null.` that can safely be ignored.
+Take note of the port on which it was started as this needs to match later configuration.
-### 4. Start the Gateway server
+##### 3. Start the gateway server
java -jar bin/server.jar
Take note of the port identified in the logging output as you will need this for accessing the gateway.
-The server will prompt you for the master secret (password). This secret is used to secure artifacts used to secure artifacts used by the gateway server for things like SSL, credential/password aliasing. This secret will have to be entered at startup unless you choose to persist it. Remember this secret and keep it safe. It represents the keys to the kingdom. See the Persisting the Master section for more information.
-
-### 5. Configure the Gateway with the topology of your Hadoop cluster
+The server will prompt you for the master secret (password).
+This secret is used to secure artifacts used to secure artifacts used by the gateway server for things like SSL, credential/password aliasing.
+This secret will have to be entered at startup unless you choose to persist it.
+Remember this secret and keep it safe.
+It represents the keys to the kingdom. See the Persisting the Master section for more information.
+
+##### 4. Configure the Gateway with the topology of your Hadoop cluster
+
+Edit the file `{GATEWAY_HOME}/deployments/sandbox.xml`
+
+Change the host and port in the urls of the `<service>` elements for WEBHDFS, WEBHCAT, OOZIE, WEBHBASE and HIVE services to match your Hadoop cluster deployment.
+
+The default configuration contains the LDAP URL for a LDAP server.
+By default that file is configured to access the demo ApacheDS based LDAP
+server and its default configuration. By default, this server listens on port 33389.
+Optionally, you can change the LDAP URL for the LDAP server to be used for authentication.
+This is set via the main.ldapRealm.contextFactory.url property in the `<gateway><provider><authentication>` section.
+
+Save the file.
+The directory `{GATEWAY_HOME}/deployments` is monitored by the gateway server.
+When a new or changed cluster topology descriptor is detected, it will provision the endpoints for the services described in the topology descriptor.
+Note that the name of the file excluding the extension is also used as the path for that cluster in the URL.
+For example the `sandbox.xml` file will result in gateway URLs of the form `http://{gateway-host}:{gateway-port}/gateway/sandbox/webhdfs`.
-Edit the file {{\{GATEWAY_HOME\}/deployments/sample.xml}}
-
-Change the host and port in the urls of the {{<service>}} elements for NAMENODE, TEMPLETON and OOZIE services to match your Hadoop cluster
-deployment.
-
-The default configuration contains the LDAP URL for a LDAP server. By default that file is configured to access the demo ApacheDS based LDAP
-server and its default configuration. By default, this server listens on port 33389. Optionally, you can change the LDAP URL for the LDAP server to be used for authentication. This is set via the main.ldapRealm.contextFactory.url property in the {{<gateway><provider><authentication>}} section.
-
-Save the file. The directory {{\{GATEWAY_HOME\}/deployments}} is monitored by the Gateway server and reacts to the discovery of a new or changed cluster topology descriptor by provisioning the endpoints and required filter chains to serve the needs of each cluster as described by the topology file. Note that the name of the file excluding the extension is also used as the path for that cluster in the URL. So for example
-the sample.xml file will result in Gateway URLs of the form {{\[http://\]}}{{{}{gateway-host\}:\{gateway-port\}/gateway/sample/namenode/api/v1}}
-
-### 6. Test the installation and configuration of your Gateway
+##### 5. Test the installation and configuration of your Gateway
Invoke the LISTSATUS operation on HDFS represented by your configured NAMENODE by using your web browser or curl:
curl -i -k -u bob:bob-password -X GET \
- 'https://localhost:8443/gateway/sample/namenode/api/v1/?op=LISTSTATUS'
+ 'https://localhost:8443/gateway/sandbox/webhdfs/v1/?op=LISTSTATUS'
The results of the above command should result in something to along the lines of the output below. The exact information returned is subject to the content within HDFS in your Hadoop cluster.
@@ -85,57 +217,50 @@ The results of the above command should
For additional information on WebHDFS, Templeton/WebHCat and Oozie REST APIs, see the following URLs respectively:
-* WebHDFS - [http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html]
-* Templeton/WebHCat - [http://people.apache.org/~thejas/templeton_doc_v1/]
-* Oozie - [http://oozie.apache.org/docs/3.3.1/WebServicesAPI.html]
-
-
-### Examples
+* WebHDFS - http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html
+* WebHCat (Templeton) - http://people.apache.org/~thejas/templeton_doc_v1
+* Oozie - http://oozie.apache.org/docs/3.3.1/WebServicesAPI.html
+* Stargate (HBase) - http://wiki.apache.org/hadoop/Hbase/Stargate
-More examples can be found [here|Examples].
+### More Examples ###
+These examples provide more detail about how to access various Apache Hadoop services via the Apache Knox Gateway.
-###. Persisting the Master Secret
+* [WebHDFS](#WebHDFS+Examples)
+* [WebHCat/Templeton](#WebHCat+Examples)
+* [Oozie](#Oozie+Examples)
+* [HBase](#HBase+Examples)
+* [Hive](#Hive+Examples)
-The master secret is required to start the server.
-This secret is used to access secured artifacts by the gateway instance.
-Keystore, trust stores and credential stores are all protected with the master secret.
-You may persist the master secret by supplying the *\-persist-master* switch at startup.
-This will result in a warning indicating that persisting the secret is less secure than providing it at startup.
-We do make some provisions in order to protect the persisted password.
+{{Sandbox Configuration}}
+-------------------------
-It is encrypted with AES 128 bit encryption and where possible the file permissions are set to only be accessible by the user that the gateway is running as.
+This version of the Apache Knox Gateway is tested against [Hortonworks Sandbox 2.x|sandbox]
-After persisting the secret, ensure that the file at config/security/master has the appropriate permissions set for your environment.
-This is probably the most important layer of defense for master secret.
-Do not assume that the encryption if sufficient protection.
+Currently there is an issue with Sandbox that prevents it from being easily used with the gateway.
+In order to correct the issue, you can use the commands below to login to the Sandbox VM and modify the configuration.
+This assumes that the name sandbox is setup to resolve to the Sandbox VM.
+It may be necessary to use the IP address of the Sandbox VM instead.
+*This is frequently but not always `192.168.56.101`.*
-A specific user should be created to run the gateway this will protect a persisted master file.
+ ssh root@sandbox
+ cp /usr/lib/hadoop/conf/hdfs-site.xml /usr/lib/hadoop/conf/hdfs-site.xml.orig
+ sed -e s/localhost/sandbox/ /usr/lib/hadoop/conf/hdfs-site.xml.orig > /usr/lib/hadoop/conf/hdfs-site.xml
+ shutdown -r now
-### Mapping Gateway URLs to Hadoop cluster URLs
+In addition to make it very easy to follow along with the samples for the gateway you can configure your local system to resolve the address of the Sandbox by the names `vm` and `sandbox`.
+The IP address that is shown below should be that of the Sandbox VM as it is known on your system.
+*This will likely, but not always, be `192.168.56.101`.*
-The Gateway functions much like a reverse proxy.
-As such it maintains a mapping of URLs that are exposed externally by the Gateway to URLs that are provided by the Hadoop cluster.
-Examples of mappings for the NameNode and Templeton are shown below.
-These mapping are generated from the combination of the Gateway configuration file (i.e. {{\{GATEWAY_HOME\}/conf/gateway-site.xml}}) and the cluster topology descriptors (e.g. {{\{GATEWAY_HOME\}/deployments/\{cluster-name\}.xml}}).
+On Linux or Macintosh systems add a line like this to the end of the file `/etc/hosts` on your local machine, *not the Sandbox VM*.
+_Note: The character between the 192.168.56.101 and vm below is a *tab* character._
-* HDFS (NameNode)
- * Gateway: {nolink:http://\{gateway-host\}:\{gateway-port\}/\{gateway-path\}/\{cluster-name\}/namenode/api/v1}
- * Cluster: {nolink:http://\{namenode-host\}:50070/webhdfs/v1}
-* WebHCat (Templeton)
- * Gateway: {nolink:http://\{gateway-host\}:\{gateway-port\}/\{gateway-path\}/\{cluster-name\}/templeton/api/v1}
- * Cluster: {nolink:http://\{templeton-host\}:50111/templeton/v1}
-* Oozie
- * Gateway: {nolink:http://\{gateway-host\}:\{gateway-port\}/\{gateway-path\}/\{cluster-name\}/oozie/api/v1}
- * Cluster: {nolink:http://\{templeton-host\}:11000/oozie/v1}
+ 192.168.56.101 vm sandbox
-The values for {{\{gateway-host\}}}, {{\{gateway-port\}}}, {{\{gateway-path\}}} are provided via the Gateway configuration file (i.e. `{GATEWAY_HOME\}/conf/gateway-site.xml`).
+On Windows systems a similar but different mechanism can be used. On recent
+versions of windows the file that should be modified is `%systemroot%\system32\drivers\etc\hosts`
-The value for {{\{cluster-name\}}} is derived from the name of the cluster topology descriptor (e.g. {{\{GATEWAY_HOME\}/deployments/\{cluster-name\}.xml}}).
-The value for {{\{namenode-host\}}} and {{\{templeton-host\}}} is provided via the cluster topology descriptor (e.g. {{\{GATEWAY_HOME\}/deployments/\{cluster-name\}.xml}}).
-Note: The ports 50070, 50111 and 11000 are the defaults for NameNode, Templeton and Oozie respectively.
-Their values can also be provided via the cluster topology descriptor if your Hadoop cluster uses different ports.
Added: incubator/knox/trunk/books/0.3.0/book_service-details.md
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/0.3.0/book_service-details.md?rev=1526638&view=auto
==============================================================================
--- incubator/knox/trunk/books/0.3.0/book_service-details.md (added)
+++ incubator/knox/trunk/books/0.3.0/book_service-details.md Thu Sep 26 18:21:04 2013
@@ -0,0 +1,29 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+--->
+
+{{Service Details}}
+-------------------
+
+TODO
+
+<<service_webhdfs.md>>
+<<service_webhcat.md>>
+<<service_oozie.md>>
+<<service_hbase.md>>
+<<service_hive.md>>
+
+
Modified: incubator/knox/trunk/books/0.3.0/config.md
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/0.3.0/config.md?rev=1526638&r1=1526637&r2=1526638&view=diff
==============================================================================
--- incubator/knox/trunk/books/0.3.0/config.md (original)
+++ incubator/knox/trunk/books/0.3.0/config.md Thu Sep 26 18:21:04 2013
@@ -45,6 +45,25 @@ A number of useful, more fine loggers ar
TODO
+### Persisting the Master Secret ###
+
+The master secret is required to start the server.
+This secret is used to access secured artifacts by the gateway instance.
+Keystore, trust stores and credential stores are all protected with the master secret.
+
+You may persist the master secret by supplying the *\-persist-master* switch at startup.
+This will result in a warning indicating that persisting the secret is less secure than providing it at startup.
+We do make some provisions in order to protect the persisted password.
+
+It is encrypted with AES 128 bit encryption and where possible the file permissions are set to only be accessible by the user that the gateway is running as.
+
+After persisting the secret, ensure that the file at config/security/master has the appropriate permissions set for your environment.
+This is probably the most important layer of defense for master secret.
+Do not assume that the encryption if sufficient protection.
+
+A specific user should be created to run the gateway this will protect a persisted master file.
+
+
### Management of Security Artifacts ###
There are a number of artifacts that are used by the gateway in ensuring the security of wire level communications, access to protected resources and the encryption of sensitive data.
Copied: incubator/knox/trunk/books/0.3.0/config_authn.md (from r1526490, incubator/knox/trunk/books/0.3.0/authn.md)
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/0.3.0/config_authn.md?p2=incubator/knox/trunk/books/0.3.0/config_authn.md&p1=incubator/knox/trunk/books/0.3.0/authn.md&r1=1526490&r2=1526638&rev=1526638&view=diff
==============================================================================
--- incubator/knox/trunk/books/0.3.0/authn.md (original)
+++ incubator/knox/trunk/books/0.3.0/config_authn.md Thu Sep 26 18:21:04 2013
@@ -15,7 +15,7 @@
limitations under the License.
--->
-### {{Authentication}} ###
+### Authentication ###
#### LDAP Configuration ####
Copied: incubator/knox/trunk/books/0.3.0/config_authz.md (from r1526490, incubator/knox/trunk/books/0.3.0/authz.md)
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/0.3.0/config_authz.md?p2=incubator/knox/trunk/books/0.3.0/config_authz.md&p1=incubator/knox/trunk/books/0.3.0/authz.md&r1=1526490&r2=1526638&rev=1526638&view=diff
==============================================================================
--- incubator/knox/trunk/books/0.3.0/authz.md (original)
+++ incubator/knox/trunk/books/0.3.0/config_authz.md Thu Sep 26 18:21:04 2013
@@ -15,7 +15,7 @@
limitations under the License.
--->
-### {{Authorization}} ###
+### Authorization ###
#### Service Level Authorization ####
Copied: incubator/knox/trunk/books/0.3.0/service_hbase.md (from r1526490, incubator/knox/trunk/books/0.3.0/hbase.md)
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/0.3.0/service_hbase.md?p2=incubator/knox/trunk/books/0.3.0/service_hbase.md&p1=incubator/knox/trunk/books/0.3.0/hbase.md&r1=1526490&r2=1526638&rev=1526638&view=diff
==============================================================================
--- incubator/knox/trunk/books/0.3.0/hbase.md (original)
+++ incubator/knox/trunk/books/0.3.0/service_hbase.md Thu Sep 26 18:21:04 2013
@@ -15,7 +15,7 @@
limitations under the License.
--->
-### {{HBase}} ###
+### HBase ###
TODO
@@ -23,7 +23,7 @@ TODO
TODO
-#### {{HBase Examples}} ####
+#### HBase Examples ####
TODO
Copied: incubator/knox/trunk/books/0.3.0/service_hive.md (from r1526490, incubator/knox/trunk/books/0.3.0/hive.md)
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/0.3.0/service_hive.md?p2=incubator/knox/trunk/books/0.3.0/service_hive.md&p1=incubator/knox/trunk/books/0.3.0/hive.md&r1=1526490&r2=1526638&rev=1526638&view=diff
==============================================================================
--- incubator/knox/trunk/books/0.3.0/hive.md (original)
+++ incubator/knox/trunk/books/0.3.0/service_hive.md Thu Sep 26 18:21:04 2013
@@ -15,7 +15,7 @@
limitations under the License.
--->
-### {{Hive}} ###
+### Hive ###
TODO
@@ -23,7 +23,7 @@ TODO
TODO
-#### {{Hive Examples}} ####
+#### Hive Examples ####
This guide provides detailed examples for how to to some basic interactions with Hive via the Apache Knox Gateway.
Copied: incubator/knox/trunk/books/0.3.0/service_oozie.md (from r1526490, incubator/knox/trunk/books/0.3.0/oozie.md)
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/0.3.0/service_oozie.md?p2=incubator/knox/trunk/books/0.3.0/service_oozie.md&p1=incubator/knox/trunk/books/0.3.0/oozie.md&r1=1526490&r2=1526638&rev=1526638&view=diff
==============================================================================
--- incubator/knox/trunk/books/0.3.0/oozie.md (original)
+++ incubator/knox/trunk/books/0.3.0/service_oozie.md Thu Sep 26 18:21:04 2013
@@ -15,7 +15,7 @@
limitations under the License.
--->
-### {{Oozie}} ###
+### Oozie ###
TODO
@@ -23,7 +23,7 @@ TODO
TODO
-#### {{Oozie Examples}} ####
+#### Oozie Examples ####
TODO
Copied: incubator/knox/trunk/books/0.3.0/service_webhcat.md (from r1526490, incubator/knox/trunk/books/0.3.0/webhcat.md)
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/0.3.0/service_webhcat.md?p2=incubator/knox/trunk/books/0.3.0/service_webhcat.md&p1=incubator/knox/trunk/books/0.3.0/webhcat.md&r1=1526490&r2=1526638&rev=1526638&view=diff
==============================================================================
--- incubator/knox/trunk/books/0.3.0/webhcat.md (original)
+++ incubator/knox/trunk/books/0.3.0/service_webhcat.md Thu Sep 26 18:21:04 2013
@@ -15,7 +15,7 @@
limitations under the License.
--->
-### {{WebHCat}} ###
+### WebHCat ###
TODO
@@ -23,7 +23,7 @@ TODO
TODO
-#### {{WebHCat Examples}} ####
+#### WebHCat Examples ####
TODO
Copied: incubator/knox/trunk/books/0.3.0/service_webhdfs.md (from r1526490, incubator/knox/trunk/books/0.3.0/webhdfs.md)
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/0.3.0/service_webhdfs.md?p2=incubator/knox/trunk/books/0.3.0/service_webhdfs.md&p1=incubator/knox/trunk/books/0.3.0/webhdfs.md&r1=1526490&r2=1526638&rev=1526638&view=diff
==============================================================================
--- incubator/knox/trunk/books/0.3.0/webhdfs.md (original)
+++ incubator/knox/trunk/books/0.3.0/service_webhdfs.md Thu Sep 26 18:21:04 2013
@@ -15,7 +15,7 @@
limitations under the License.
--->
-### {{WebHDFS}} ###
+### WebHDFS ###
TODO
Modified: incubator/knox/trunk/books/common/header.md
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/common/header.md?rev=1526638&r1=1526637&r2=1526638&view=diff
==============================================================================
--- incubator/knox/trunk/books/common/header.md (original)
+++ incubator/knox/trunk/books/common/header.md Thu Sep 26 18:21:04 2013
@@ -18,6 +18,9 @@
<link href="book.css" rel="stylesheet"/>
[asl]: http://www.apache.org/licenses/LICENSE-2.0
+[sandbox]: http://hortonworks.com/products/hortonworks-sandbox
+[mirror]: http://www.apache.org/dyn/closer.cgi/incubator/knox
+
[y]: check.png "Yes"
[n]: error.png "No"
[?]: question.png "Unknown"
Modified: incubator/knox/trunk/markbook/src/main/java/org/apache/hadoop/gateway/markbook/MarkBook.java
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/markbook/src/main/java/org/apache/hadoop/gateway/markbook/MarkBook.java?rev=1526638&r1=1526637&r2=1526638&view=diff
==============================================================================
--- incubator/knox/trunk/markbook/src/main/java/org/apache/hadoop/gateway/markbook/MarkBook.java (original)
+++ incubator/knox/trunk/markbook/src/main/java/org/apache/hadoop/gateway/markbook/MarkBook.java Thu Sep 26 18:21:04 2013
@@ -31,6 +31,7 @@ import org.pegdown.Extensions;
import org.pegdown.PegDownProcessor;
import java.io.File;
+import java.io.FileNotFoundException;
import java.io.IOException;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
@@ -54,7 +55,9 @@ public class MarkBook {
// matcher.find();
// System.out.println( replace( matcher, text, "X" ) );
-// System.out.println( replaceIdentifiers( "{{text}}" ) );
+// System.out.println( replaceHeadings( "#text#" ) );
+// System.out.println( replaceHeadings( "# text #" ) );
+// System.out.println( replaceHeadings( "## text ##" ) );
// System.out.println( removeComments( "line\r\n<!--- \r\n comment \r\n comment \r\n---> \r\nline" ) );
CommandLine command = parseCommandLine( args );
@@ -87,7 +90,7 @@ public class MarkBook {
private static String loadMarkdown( File file ) throws IOException {
String text = FileUtils.readFileToString( file );
text = removeComments( text );
- text = replaceIdentifiers( text );
+ text = replaceHeadings( text );
text = replaceIncludes( file, text );
return text;
}
@@ -102,21 +105,25 @@ public class MarkBook {
String includeString = loadMarkdown( includeFile );
text = replace( matcher, text, includeString );
} else {
- text = replace( matcher, text, includeFileName );
+ throw new FileNotFoundException( includeFile.getAbsolutePath() );
+ //text = replace( matcher, text, includeFileName );
}
matcher = pattern.matcher( text );
}
return text;
}
- private static String replaceIdentifiers( String text ) throws IOException {
- Pattern pattern = Pattern.compile( "\\{\\{(.+?)\\}\\}" );
+ private static String replaceHeadings( String text ) throws IOException {
+ Pattern pattern = Pattern.compile( "^(#+)(.+?)#*$", Pattern.MULTILINE );
Matcher matcher = pattern.matcher( text );
while( matcher.find() ) {
- String name = matcher.group( 1 ).trim();
+ String tag = matcher.group( 1 );
+ String name = matcher.group( 2 ).trim();
String id = name.replaceAll( "\\s", "+" );
- text = replace( matcher, text, String.format( "<a id=\"%s\"></a>%s", id, name ) );
- matcher = pattern.matcher( text );
+ if( !name.startsWith( "<a id=" ) ) {
+ text = replace( matcher, text, String.format( "%s <a id=\"%s\"></a>%s %s", tag, id, name, tag ) );
+ matcher = pattern.matcher( text );
+ }
}
return text;
}