You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@slider.apache.org by st...@apache.org on 2014/06/30 17:37:10 UTC
[21/50] [abbrv] SLIDER-121 removed site documentation from git source
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/manpage.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/manpage.md b/src/site/markdown/manpage.md
deleted file mode 100644
index 351fcc6..0000000
--- a/src/site/markdown/manpage.md
+++ /dev/null
@@ -1,483 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# Apache Slider: YARN-hosted applications
-
-## NAME
-
-slider -YARN-hosted applications
-
-## SYNOPSIS
-
-Slider enables applications to be dynamically created on a YARN-managed datacenter.
-The program can be used to create, pause, and shutdown the application. It can also be used to list current active
-and existing but not running "frozen" application instances.
-
-## CONCEPTS
-
-1. A *Slider application* is an application packaged to be deployed by Slider. It consists of one or more distributed *components*
-
-1. A *Slider application instance* is a slider application configured to be deployable on a specific YARN cluster, with a specific configuration. An instance can be *live* -actually running- or *frozen*. When frozen all its configuration details and instance-specific data are preserved on HDFS.
-
-1. An *image* is a *tar.gz* file containing binaries used to create the application. 1. Images are kept in the HDFS filesystem and identified by their path names; filesystem permissions can be used to share images amongst users.
-
-1. An *image configuration* is a directory that is overlaid file-by-file onto the conf/ directory inside the HBase image.
-
-1. Users can have multiple image configurations -they too are kept in HDFS, identified by their path names, and can be shared by setting the appropriate permissions, along with a configuration template file.
-
-1. Only those files provided in the image configuration directory overwrite the default values contained in the image; all other configuration files are retained.
-
-1. Late-binding properties can also be provided at create time.
-
-1. Slider can overwrite some of the configuration properties to enable the dynamically created components to bind correctly to each other.
-
-1. An *instance directory* is a directory created in HDFS describing the application instance; it records the configuration -both user specified, application-default and any dynamically created by slider.
-
-1. A user can create an application instance.
-
-1. A live instances can be *frozen*, saving its final state to its application instance state directory. All running components are shut down.
-
-1. A frozen instance can be *thawed* -a its components started on or near the servers where they were previously running.
-
-1. A frozen instance can be *destroyed*.
-
-1. Running instances can be listed.
-
-1. An instance consists of a set of components
-
-1. The supported component types depends upon the slider application.
-
-1. the count of each component must initially be specified when a application instance is created.
-
-1. Users can flex an application instance: adding or removing components dynamically.
-If the application instance is live, the changes will have immediate effect. If not, the changes will be picked up when the instance is next thawed.
-
-
-<!--- ======================================================================= -->
-
-## Invoking Slider
-
-
- slider <ACTION> [<name>] [<OPTIONS>]
-
-
-<!--- ======================================================================= -->
-
-## COMMON COMMAND-LINE OPTIONS
-
-### `--conf configuration.xml`
-
-Configure the Slider client. This allows the filesystem, zookeeper instance and other properties to be picked up from the configuration file, rather than on the command line.
-
-Important: *this configuration file is not propagated to the application. It is purely for configuring the client itself.
-
-### `-D name=value`
-
-Define a Hadoop configuration option which overrides any options in the client configuration XML files.
-
-
-### `-m, --manager url`
-
-URL of the YARN resource manager
-
-
-### `--fs filesystem-uri`
-
-Use the specific filesystem URI as an argument to the operation.
-
-
-
-
-
-<!--- ======================================================================= -->
-
-
-<!--- ======================================================================= -->
-
-## Actions
-
-COMMANDS
-
-
-
-### `build <name>`
-
-Build an instance of the given name, with the specific options.
-
-It is not started; this can be done later with a `thaw` command.
-
-### `create <name>`
-
-Build and run an applicaton instance of the given name
-
-The `--wait` parameter, if provided, specifies the time to wait until the YARN application is actually running. Even after the YARN application has started, there may be some delay for the instance to start up.
-
-### Arguments for `build` and `create`
-
-
-
-##### `--package <uri-to-package>`
-
-This define the slider application package to be deployed.
-
-
-##### `--option <name> <value>`
-
-Set a application instance option.
-
-Example:
-
-Set an option to be passed into the `-site.xml` file of the target system, reducing
-the HDFS replication factor to 2. (
-
- --option site.dfs.blocksize 128m
-
-Increase the number of YARN containers which must fail before the Slider application instance
-itself fails.
-
- -O slider.container.failure.threshold 16
-
-##### `--appconf dfspath`
-
-A URI path to the configuration directory containing the template application specification. The path must be on a filesystem visible to all nodes in the YARN cluster.
-
-1. Only one configuration directory can be specified.
-1. The contents of the directory will only be read when the application instance is created/built.
-
-Example:
-
- --appconf hdfs://namenode/users/slider/conf/hbase-template
- --appconf file://users/accumulo/conf/template
-
-
-
-##### `--apphome localpath`
-
-A path to the home dir of a pre-installed application. If set when a Slider
-application instance is created, the instance will run with the binaries pre-installed
-on the nodes at this location
-
-*Important*: this is a path in the local filesystem which must be present
-on all hosts in the YARN cluster
-
-Example
-
- --apphome /usr/hadoop/hbase
-
-##### `--template <filename>`
-
-Filename for the template application instance configuration. This
-will be merged with -and can overwrite- the built-in configuration options, and can
-then be overwritten by any command line `--option` and `--compopt` arguments to
-generate the final application configuration.
-
-##### `--resources <filename>`
-
-Filename for the resources configuration. This
-will be merged with -and can overwrite- the built-in resource options, and can
-then be overwritten by any command line `--resopt`, `--rescompopt` and `--component`
-arguments to generate the final resource configuration.
-
-
-##### `--image path`
-
-The full path in Hadoop HDFS to a `.tar` or `.tar.gz` file containing
-the binaries needed to run the target application.
-
-Example
-
- --image hdfs://namenode/shared/binaries/hbase-0.96.tar.gz
-
-##### `--component <name> <count>`
-
-The desired number of instances of a component
-
-
-Example
-
- --component worker 16
-
-This just sets the `component.instances` value of the named component's resource configuration.
-it is exactly equivalent to
-
- --rco worker component.instances 16
-
-
-
-#### `--compopt <component> <option> <value>`
-
-Provide a specific application configuration option for a component
-
-Example
-
- --compopt master env.TIMEOUT 10000
-
-These options are saved into the `app_conf.json` file; they are not used to configure the YARN Resource
-allocations, which must use the `--rco` parameter
-
-#### Resource Component option `--rescompopt` `--rco`
-
-`--rescompopt <component> <option> <value>`
-
-Set any role-specific option, such as its YARN memory requirements.
-
-Example
-
- --rco worker master yarn.memory 2048
- --rco worker worker yarn.memory max
-
-
-##### `--zkhosts host1:port1,[host2:port2,host3:port3, ...] `
-
-The zookeeper quorum.
-
-Example
-
- --zkhosts zk1:2181,zk2:2181,zk3:4096
-
-If unset, the zookeeper quorum defined in the property `slider.zookeeper.quorum`
-is used
-
-### `destroy <name>`
-
-Destroy a (stopped) applicaton instance .
-
-Important: This deletes all persistent data
-
-Example
-
- slider destroy instance1
-
-### `exists <name> [--live]`
-
-Probe the existence of the named Slider application instance. If the `--live` flag is set, the instance
-must actually be running
-
-If not, an error code is returned.
-
-When the --live` flag is unset, the command looks for the application instance to be
-defined in the filesystem -its operation state is not checked.
-
-Return codes
-
- 0 : application instance is defined in the filesystem
- 70 : application instance is unknown
-
-Example:
-
- slider exists instance4
-
-#### Live Tests
-
-When the `--live` flag is set, the application instance must be running for the command
-to succeed
-
-1. The probe does not check the status of any Slider-deployed services, merely that a application instance has been deployed
-1. A application instance that is finished or failed is not considered to be live.
-
-Return codes
-
- 0 : application instance is running
- -1 : application instance exists but is not running
- 70 : application instance is unknown
-
-
-Example:
-
- slider exists instance4 --live
-
-### `flex <name> [--component component count]* `
-
-Flex the number of workers in an application instance to the new value. If greater than before, new copies of the component will be requested. If less, component instances will be destroyed.
-
-This operation has a return value of 0 if the size of a running instance was changed.
-
-It returns -1 if there is no running application instance, or the size of the flexed instance matches that of the original -in which case its state does not change.
-
-Example
-
- slider flex instance1 --component worker 8 --filesystem hdfs://host:port
- slider flex instance1 --component master 2 --filesystem hdfs://host:port
-
-
-### `freeze <name> [--force] [--wait time] [--message text]`
-
-freeze the application instance. The running application is stopped. Its settings are retained in HDFS.
-
-The `--wait` argument can specify a time in seconds to wait for the application instance to be frozen.
-
-The `--force` flag causes YARN asked directly to terminate the application instance.
-The `--message` argument supplies an optional text message to be used in
-the request: this will appear in the application's diagnostics in the YARN RM UI.
-
-If an unknown (or already frozen) application instance is named, no error is returned.
-
-Examples
-
- slider freeze instance1 --wait 30
- slider freeze instance2 --force --message "maintenance session"
-
-
-### `list <name>`
-
-List running Slider application instances visible to the user.
-
-If an instance name is given and there is no running instance with that name, an error is returned.
-
-Example
-
- slider list
- slider list instance1
-
-### `registry (--list | --listconf | --getconf <conf>) [--name <name>] [--servicetype <servicetype>] [--verbose]`
-
-List registered application instances visible to the user. This is slightly
-different from the `slider list` command in that it does not make use of the
-YARN application list. Instead it communicates with Zookeeper -and works
-with any applications which has registered itself with the "service registry"
-
-The `--name <name>` option names the registry entry to work with. For slider applications,
-this is the application instance
-
-
-The `--servicetype <servicetype>` option allows a different service type to be chosen.
-The default is `"org.apache.slider`
-
-The `--verbose` flag triggers more verbose output on the operations
-
-The `--internal` flag indicates the configurations to be listed and retrieved
-are from the "internal" list of configuration data provided for use within a
-deployed application.
-
-There are two common exit codes, the exact values being documented
-in [Exit Codes](exitcodes.md)
-
-1. If there is no matching service then the operation fails with the
-`EXIT_NOT_FOUND` code (77).
-2. If there are no configurations in a listing, or the named configuration
-is not found, the command returns the exit code `EXIT_NOT_FOUND` (77)
-
-Operations:
-
-#### `registry --list [--servicetype <servicetype>] [--name <name>] [--verbose]`
-
-List all services of the service type and optionally the name.
-
-
-
-#### `registry --listconf [--name <name>] [--internal] [--servicetype <servicetype>]`
-
-List the configurations exported by of a named application
-
-
-
-#### `registry --getconf <configuration> [--format (xml|json|properties)] [--dest <path>] [--internal] ` get the configuration
-
-Get a named configuration in a chosen format. Default: XML
-
- `--dest <path>` : the filename or directory to save a configuration to.
- `--format (xml|json|properties)` defines the output format
-
-
-
-
-
-
-
-### `status <name> [--out <filename>]`
-
-Get the status of the named application instance in JSON format. A filename can be used to
-specify the destination.
-
-Examples:
-
- slider status instance1 --manager host:port
-
- slider status instance2 --manager host:port --out status.json
-
-
-
-### `thaw <name> [--wait time`]
-
-Resume a frozen application instance, recreating it from its previously saved state. This will include a best-effort attempt to create the same number of nodes as before, though their locations may be different.
-
-Examples:
-
- slider thaw instance2
- slider thaw instance1 --wait 60
-
-
-If the application instance is already running, this command will not affect it.
-
-
-### `version`
-
-The command `slider version` prints out information about the compiled
-Slider application, the version of Hadoop against which it was built -and
-the version of Hadoop that is currently on its classpath.
-
-Note that this is the client-side Hadoop version, not that running on the server, though
-that can be obtained in the status operation
-
-
-
-## Commands for testing
-
-
-These are clearly abnormal operations; they are here primarily for testing
--and documented for completeness.
-
-### `kill-container <name> --id container-id`
-
-Kill a YARN container belong to the application. This is useful primarily for testing the
-resilience to failures.
-
-Container IDs can be determined from the application instance status JSON document.
-
-
-### `am-suicide <name> [--exitcode code] [--message message] [--wait time]`
-
-This operation is purely for testing Slider Application Master restart;
-it triggers an asynchronous self-destruct operation in the AM -an
-operation that does not make any attempt to cleanly shut down the process.
-
-If the application has not exceeded its restart limit (as set by
-`slider.yarn.restart.limit`), YARN will attempt to restart the failed application.
-
-Example
-
- slider am-suicide --exitcode 1 --wait 5000 -message "test"
-
-<!--- ======================================================================= -->
-
-
-## Instance Naming
-
-Application instance names must:
-
-1. be at least one character long
-1. begin with a lower case letter
-1. All other characters must be in the range \[a-z,0-9,_]
-1. All upper case characters are converted to lower case
-
-Example valid names:
-
- slider1
- storm4
- hbase_instance
- accumulo_m1_tserve4
-
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/registry/a_YARN_service_registry.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/registry/a_YARN_service_registry.md b/src/site/markdown/registry/a_YARN_service_registry.md
deleted file mode 100644
index b695106..0000000
--- a/src/site/markdown/registry/a_YARN_service_registry.md
+++ /dev/null
@@ -1,226 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# A YARN Service Registry for Apache Slider
-
-## April 2014
-
-# Introduction
-
-This document looks at the needs and options of a service registry.
-
-The core issue is that as the location(s) of a dynamically deployed application are unknown, the standard Hadoop and Java configuration model of some form of text files containing hostnames, ports and URLS no longer works. You cannot define up-front where a service will be.
-
-Some Hadoop applications -HBase and Accumulo -have solved this with custom ZK bindings. This works for the specific clients, but requires hbase and accumulo client JARs in order to be able to work with the content. (or a re-implementation with knowledge of the non-standard contents of the ZK nodes)
-
-Other YARN applications will need to publish their bindings - this includes, but is not limited to- Slider deployed applications. Again, these applications can use their own registration and binding model, which would again require custom clients to locate the registry information and parse the contents.
-
-YARN provides some minimal publishing of AM remote endpoints: a URL to what is assumed to be a Web UI (not a REST API), and an IPC port. The URL is displayed in the YARN UI -in which case it is accessed via a proxy which (currently) only support HTTP GET operations. The YARN API call to list all applications can be used to locate a named instance of an application by (user, application-type, name), and then obtain the raw URL and IPC endpoints. This enumeration process is an O(apps) operation on the YARN RM and only provides access to those two endpoints. Even with the RAW URL, REST operations have proven "troublesome", due to a web filter which redirects all direct requests to the proxy -unless it comes from the same host as the proxy.
-
-Hadoop client applications tend to retrieve all their configuration information from files in the local filesystem, hadoop-site.xml, hdfs-site.xml, hbase-site.xml, etc. This requires the configuration files to be present on all systems. Tools such as Ambari can keep the files in the server up to date -assuming a low rate of change- ---but these tools do nothing for the client applications themselves. It is up to the cluster clients to (somehow) retrieve these files, and to keep their copies up to date. *This is a problem that exists with today's applications*.
-
-As an example, if a YARN client does not know the value of "yarn.application.classpath", it cannot successfully deploy any application in the YARN cluster which needs the cluster-side Hadoop and YARN JARs on its application master's classpath. This is not a theoretical problem, as some clusters have a different classpath from the default: without a correct value the Slider AM does not start. And, as it is designed to be run remotely, it cannot rely on a local installation of YARN to provide the correct values ([YARN-973](https://issues.apache.org/jira/browse/YARN-973)).
-
-# What do we need?
-
-**Discovery**: An IPC and URL discovery system for service-aware applications to use to look up a service to which it wishes to talk to. This is not an ORB -it's not doing redirection -, but it is something that needs to be used before starting IPC or REST communications.
-
-**Configuration**: A way for clients of a service to retrieve more configuration data than simply the service endpoints. For example: everything needed to create a site.xml document.
-
-## Client-side
-
-* Allow clients of a YARN application to locate the service instance and its service ports (web, IPC, REST...) efficiently even on a large YARN cluster.
-
-* Allow clients to retrieve configuration values which can be processed client-side into the configuration files and options which the application needs
-
-* Give clients confidence that the service with which they interact is the one they expect to interact with -not another potentially malicious service deployed by a different user.
-
-* clients to be able to watch a service and retrieve notification of changes
-
-* cross-language support.
-
-## For all Services
-
-* Allow services to publish their binding details for the AM and of code running in the containers (which may be published by the containers)
-
-* Use entries in registry as a way of enforcing uniqueness of the instance (app, owner, name)?
-
-* values to update when a service is restarted on a different host
-
-* values to indicate when a service is not running. This may be implicit "no entry found" or explicit "service exists but not running"
-
-* Services to be able to act as clients to other services
-
-## For Slider Services (and presumably others)
-
-* Ability to publish information about configuration documents that can be retrieved -and URLs
-
-* Ability to publish facts internal to the application (e.g. agent reporting URLs)
-
-* Ability to use service paths as a way to ensure a single instance of a named service can be deployed by a user
-
-## Management and PaaS UIs
-
-* Retrieve lists of web UI URLs of AM and of deployed components
-
-* Enum components and their status
-
-* retrieve dynamic assignments of IPC ports
-
-* retrieve dynamic assignments of JMX ports
-
-* retrieve any health URLs for regular probes
-
-* Listen to changes in the service mix -the arrival and departure of service instances, as well as changes in their contents.
-
-
-
-## Other Needs
-
-* Registry-configured applications. In-cluster applications should be able to subscribe to part of the registry
-to pick up changes that affect them -both for their own application configuration, and for details about
-applications on which they depend themselves.
-
-* Knox: get URLs that need to be converted into remote paths
-
-* Cloud-based deployments: work on virtual infrastructures where hostnames are unpredictable.
-
-# Open Source Registry code
-
-What can we use to implement this from ASF and ASF-compatible code?
-
-## Zookeeper
-
-We'd need a good reason not to use this. There are still some issues
-
-1. Limits on amount of published data?
-
-2. Load limits, especially during cluster startup, or if a 500-mapper job all wants to do a lookup.
-
-3. Security story
-
-4. Impact of other ZK load on the behaviour of the service registry -will it cause problems if overloaded -and are they recoverable?
-
-## Apache Curator
-
-Netflix's core curator -now [Apache Curator](http://curator.apache.org/)- framework adds a lot to make working with ZK easier, including pluggable retry policies, binding tools and other things.
-
-There is also its "experimental" [service discovery framework](http://curator.apache.org/curator-x-discovery-server/index.html), which
-
-1. Allows a service to register a URL with a name and unique ID (and custom metadata). multiple services of a given name can be registered
-
-2. Allows a service to register >1 URL.
-
-3. Has a service client which performs lookup and can cache results.
-
-4. Has a REST API
-
-Limitations
-
-* The service discovery web UI and client does not work with the version of
-Jackson (1.8.8) in Hadoop 2.4. The upgraded version in Hadoop 2.5 is compatible [HADOOP-10104](https://issues.apache.org/jira/browse/HADOOP-10104).
-
-* The per-entry configuration payload attempts to get jason to perform Object/JSON mapping with the classname provided as an attribute in the JSON. This destroys all ability of arbitrary applications to parse the published data, as well as cross-language clients -is brittle and morally wrong from a data-sharing perspective.
-
- {
-
- "name" : "name",
- "id" : "service",
- "address" : "localhost",
- "port" : 8080,
- "sslPort" : 443,
- "payload" : {
- "@class" : "org.apache.slider.core.registry.ServiceInstanceData",
- "externalView" : {
- "key" : "value"
- }
- },
- "registrationTimeUTC" : 1397249829062,
- "serviceType" : "DYNAMIC",
- "uriSpec" : {
- "parts" : [ {
- "value" : "http:",
- "variable" : false
- }, {
- "value" : ":",
- "variable" : false
- } ]
- }
- }
-
-
-
-## [Helix Service Registry](http://helix.apache.org/0.7.0-incubating-docs/recipes/service_discovery.html)
-
-This is inside Helix somewhere, used in LI in production at scale -and worth looking at. LI separate their Helix Zookeeper Quorum from their application-layer quorum, to isolate load.
-
-Notable features
-
-1. The registry is also the liveness view of the deployed application. Client's aren't watching the service registry for changes, they are watching Helix's model of the deployed application.
-1. The deployed application can pick up changes to its state the same way, allowing for live application manipulation.
-1. Tracks nodes that continually join/leave the group and drops them as unreliable.
-
-## Twill Service Registry
-
-Twill's [service registry code](http://twill.incubator.apache.org/apidocs/index.html), lets applications register a [(hostname, port)](http://twill.incubator.apache.org/apidocs/org/apache/twill/discovery/Discoverable.html) pair in the registry by a name, a name by which clients can look up and enumerate all services with a specific name.
-
-Clients can subscribe to changes in the list of services with a specific name -so picking up the arrival and departure of instances, and probe to see if a previously discovered entity is still registered.
-
-Zookeeper- and in-memory registry implementations are provided.
-
-One nice feature about this architecture -and Twill in general- is that its general single-method callback model means that it segues nicely into Java-8 lambda-expressions. This is something to retain in a YARN-wide service registry.
-
-Comparing it to curator, it offers a proper subset of curator's registered services [ServiceInstance](http://curator.apache.org/apidocs/org/apache/curator/x/discovery/ServiceInstance.html) -implying that you could publish and retrieve Curator-registered services via a new implementation of Twill's DiscoveryService. This would require extensions to the curator service discovery client allow ZK nodes to be watched for changes. This is a feature that would be useful in many use cases -such as watching service availability across a cluster, or simply blocking until a dependent service was launched.
-
-As with curator, the amount of information that can be published isn't enough for management tools to make effective use of the service registration, while for slider there's no way to publish configuration data. However a YARN registry will inevitably be a superset of the Twill client's enumerated and retrieved data -so if its registration API were sufficient to register a minimal service, supporting the YARN registry via Twill's existing API should be straightforward.
-
-## Twitter Commons Service Registration
-
-[Twitter Commons](https://github.com/twitter/commons) has a service registration library, which allows for registration of sets of servers, [publishing the hostname and port of each](http://twitter.github.io/commons/apidocs/com/twitter/common/service/registration/package-tree.html)., along with a map of string properties.
-
-Zookeeper based, it suffices if all servers are identical and only publishing single (hostname, port) pairs for callers.
-
-## AirBnB Smartstack
-
-SmartStack is [Air BnB's cloud-based service discovery system](http://nerds.airbnb.com/smartstack-service-discovery-cloud/).
-
-It has two parts, *Nerve* and *Synapse*:
-
-[**Nerve**](https://github.com/airbnb/nerve) is a ruby agent designed to monitor processes and register healthy instances in ZK (or to a mock reporter). It includes [probes for TCP ports, HTTP and rabbitMQ](https://github.com/airbnb/nerve/tree/master/lib/nerve/service_watcher). It's [a fairly simple liveness monitor](https://github.com/airbnb/nerve/blob/master/lib/nerve/service_watcher.rb).
-
-[**Synapse**](https://github.com/airbnb/synapse) takes the data and uses it to configure [HAProxy instances](http://haproxy.1wt.eu/). HAProxy handles the load balancing, queuing and integrating liveness probes into the queues. Synapse generates all the configuration files for an instance -but also tries to reconfigure the live instances via their socket APIs,
-
-Alongside these, AirBnB have another published project on Github, [Optica](https://github.com/airbnb/optica), which is a web application for nodes to register themselves with (POST) and for others to query. It publishes events to RabbitMQ, and again uses ZK to store state.
-
-AirBnB do complain a bit about ZK and its brittleness. They do mention that they suspect it is due to bugs in the Ruby ZK client library. This may be exacerbated by in-cloud deployments. Hard-coding the list of ZK nodes may work for a physical cluster, but in a virtualized cluster, the hostnames/IP Addresses of those nodes may change -leading to a meta-discovery problem: how to find the ZK quorum -especially if you can't control the DNS servers.
-
-## [Apache Directory](http://directory.apache.org/apacheds/)
-
-This is an embeddable LDAP server
-
-* Embeddable inside Java apps
-
-* Supports Kerberos alongside X.500 auth. It can actually act as a Key server and TGT if desired.
-
-* Supports DNS and DHCP queries.
-
-* Accessible via classic LDAP APIs.
-
-This isn't a registry service directly, though LDAP queries do make enumeration of services *and configuration data* straightforward. As LDAP libraries are common across languages -even built in to the Java runtime- LDAP support makes publishing information to arbitrary clients relatively straightforward.
-
-If service information were to be published via LDAP, then it should allow IT-managed LDAP services to both host this information, and publish configuration data. This would be relevant for classic Hadoop applications if we were to move the Configuration class to support back-end configuration sources beyond XML files on the classpath.
-
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/registry/index.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/registry/index.md b/src/site/markdown/registry/index.md
deleted file mode 100644
index 148b4bc..0000000
--- a/src/site/markdown/registry/index.md
+++ /dev/null
@@ -1,47 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# Apache Slider Service Registry
-
-The service registry is a core part of the Slider Architecture -it is how
-dynamically generated configurations are published for clients to pick up.
-
-The need for a service registry goes beyond Slider, however. We effectively
-have application-specific registries for HBase and Accumulo, and explicit
-registries in Apache Helix and Apache Twill, as well as re-usable registry
-code in Apache Curator.
-
-[YARN-913](https://issues.apache.org/jira/browse/YARN-913) covers the need
-for YARN itself to have a service registry. This would be the ideal ultimate
-solution -it would operate at a fixed location/ZK path, and would be guaranteed
-to be on all YARN clusters, so code could be written expecting it to be there.
-
-It could also be used to publish binding data from static applications,
-including HBase, Accumulo, Oozie, -applications deployed by management tools.
-Unless/until these applications self-published their binding data, it would
-be the duty of the management tools to do the registration.
-
-
-
-## Contents
-
-1. [YARN Application Registration and Binding: the Problem](the_YARN_application_registration_and_binding_problem.html)
-1. [A YARN Service Registry](a_YARN_service_registry.html)
-1. [April 2014 Initial Registry Design](initial_registry_design.html)
-1. [Service Registry End-to-End Scenarios](service_registry_end_to_end_scenario.html)
-1. [P2P Service Registries](p2p_service_registries.html)
-1. [References](references.html)
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/registry/initial_registry_design.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/registry/initial_registry_design.md b/src/site/markdown/registry/initial_registry_design.md
deleted file mode 100644
index e132b39..0000000
--- a/src/site/markdown/registry/initial_registry_design.md
+++ /dev/null
@@ -1,110 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# April 2014 Initial Registry Design for Apache Slider
-
-This is the plan for the initial registry design.
-
-1. Use Apache Curator [service discovery code](http://curator.apache.org/curator-x-discovery/index.html).
-
-2. AMs to register as (user, name). Maybe "service type" if we add that as an option in the slider configs
-
-3. Lift "external view" term from Helix -concept that this is the public view, not internal.
-
-4. application/properties section to list app-wide values
-
-5. application/services section to list public service URLs; publish each as unique-ID-> (human name, URL, human text). code can resolve from UniqueID; UIs can use human data.
-
-6. String Template 2 templates for generation of output (rationale: library for Python Java, .NET)
-
-7. Java CLI to retrieve values from ZK and apply named template (local, hdfs). Include ability to restrict to list of named properties (pattern match).
-
-8. AM to serve up curator service (later -host in RM? elsewhere?)
-
-### forwards-compatilibity
-
-1. This initial design will hide the fact that Apache Curator is being used to discover services,
-by storing information in the payload, `ServiceInstanceData` rather than in (the minimdal) curator
-service entries themselves. If we move to an alternate registry, provided we
-can use the same datatype -or map to it- changes should not be visible.
-
-1. The first implementation will not support watching for changes.
-
-### Initial templates
-
-* hadoop XML conf files
-
-* Java properties file
-
-* HTML listing of services
-
-
-
-## Example Curator Service Entry
-
-This is the prototype's content
-
-Toplevel
-
- service CuratorServiceInstance{name='slider', id='stevel.test_registry_am', address='192.168.1.101', port=62552, sslPort=null, payload=org.apache.slider.core.registry.info.ServiceInstanceData@4e9af21b, registrationTimeUTC=1397574073203, serviceType=DYNAMIC, uriSpec=org.apache.curator.x.discovery.UriSpec@ef8dacf0}
-
-Slider payload.
-
- payload=
- {
- "internalView" : {
- "endpoints" : {
- "/agents" : {
- "value" : "http://stevel-8.local:62552/ws/v1/slider/agents",
- "protocol" : "http",
- "type" : "url",
- "description" : "Agent API"
- }
- },
- "settings" : { }
- },
-
- "externalView" : {
- "endpoints" : {
- "/mgmt" : {
- "value" : "http://stevel-8.local:62552/ws/v1/slider/mgmt",
- "protocol" : "http",
- "type" : "url",
- "description" : "Management API"
- },
-
- "slider/IPC" : {
- "value" : "stevel-8.local/192.168.1.101:62550",
- "protocol" : "org.apache.hadoop.ipc.Protobuf",
- "type" : "address",
- "description" : "Slider AM RPC"
- },
- "registry" : {
- "value" : "http://stevel-8.local:62552/ws/registry",
- "protocol" : "http",
- "type" : "url",
- "description" : "Registry"
- }
- },
- "settings" : { }
- }
- }
-
-
-
-
-
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/registry/p2p_service_registries.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/registry/p2p_service_registries.md b/src/site/markdown/registry/p2p_service_registries.md
deleted file mode 100644
index eaf5097..0000000
--- a/src/site/markdown/registry/p2p_service_registries.md
+++ /dev/null
@@ -1,137 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# P2P Service Registries for Apache Slider
-
-Alongside the centralized service registries, there's much prior work on
-P2P discovery systems, especially for mobile and consumer devices.
-
-They perform some multicast- or distributed hash table-based lookup,
-and tend to have common limitations:
-
-* scalability
-
-* the bootstrapping problem
-
-* security: can you trust the results to be honest?
-
-* consistency: can you trust the results to be complete and current?
-
-Bootstrapping is usually done via multicast, possibly then switching
-to unicast for better scale. As multicasting doesn't work in cloud
-infrastructures, none of the services work unmodified in public
-clouds. There's multiple anecdotes of
-[Amazon's SimpleDB service](http://aws.amazon.com/simpledb/) being used as a
-registry for in-EC2 applications. At the very least, this service and its
-equivalents in other cloud providers could be used to bootstrap ZK client
-bindings in cloud environments.
-
-## Service Location Protocol
-
-Service Location Protocol is a protocol for discovery services that came out
-of Sun, Novell and others -it is still available for printer discovery and
-suchlike
-
-It supports both a multicast discovery mechanism, and a unicast protocol
-to talk to a Directory Agent -an agent that is itself discovered by multicast
-requests, or by listening for the agent's intermittent multicast announcements.
-
-There's an extension to DHCP, RFC2610, which added the ability for DHCP to
-advertise Directory Agents -this was designed to solve the bootstrap problem
-(though not necessarily security or in-cloud deployment). Apart from a few
-mentions in Windows Server technical notes, it does not appear to exist.
-
-* [[RFC2608](http://www.ietf.org/rfc/rfc2608.txt)] *Service Location Protocol, Version 2* , IEEE, 1999
-
-* [[RFC3224](http://www.ietf.org/rfc/rfc3224.txt)] *Vendor Extensions for Service Location Protocol, Version 2*, IETF, 2003
-
-* [[RFC2610](http://www.ietf.org/rfc/rfc2610.txt)] *DHCP Options for Service Location Protocol, IETF, 1999*
-
-## [Zeroconf](http://www.zeroconf.org/)
-
-The multicast discovery service implemented in Apple's Bonjour system
---multicasting DNS lookups to all peers in the subnet.
-
-This allows for URLs and hostnames to be dynamically positioned, with
-DNS domain searches allowing for enumeration of service groups.
-
-This protocol scales very badly; the load on *every* client in the
-subnet is is O(DNS-queries-across-subnet), hence implicitly `O(devices)*O(device-activity)`.
-
-The special domains `_tcp.`, `_udp.` and their subdomains can also be
-served up via a normal DNS server.
-
-## [Jini/Apache River](http://river.apache.org/doc/specs/html/lookup-spec.html)
-
-Attribute-driven service enumeration, which drives the, Java-client-only
-model of downloading client-side code. There's no requirement for the remote
-services to be in Java, only that drivers are.
-
-## [Serf](http://www.serfdom.io/)
-
-This is a library that implements the [SWIM protocol](http://www.cs.cornell.edu/~asdas/research/dsn02-swim.pdf) to propagate information around a cluster. Apparently works in virtualized clusters too. It's already been used in a Flume-on-Hoya provider.
-
-## [Anubis](http://sourceforge.net/p/smartfrog/svn/HEAD/tree/trunk/core/components/anubis/)
-
-An HP Labs-built [High Availability tuple-space](http://sourceforge.net/p/smartfrog/svn/HEAD/tree/trunk/core/components/anubis/doc/HPL-2005-72.pdf?format=raw) in SmartFrog; used in production in some of HP's telco products. An agent publishes facts into the T-Space, and within one heartbeat all other agents have it. One heart-beat later, unless there's been a change in the membership, the publisher knows the others have it. One heartbeat later the agents know the publisher knows it, etc.
-
-Strengths:
-
-* The shared knowledge mechanism permits reasoning and mathematical proofs.
-
-* Strict ordering between heartbeats implies an ordering in receipt.
-This is stronger than ZK's guarantees.
-
-* Lets you share a moderate amount of data (the longer the heartbeat
-interval, the more data you can publish).
-
-* Provided the JVM hosting the Anubis agent is also hosting the service,
-liveness is implicit
-
-* Secure to the extent that it can be locked down to allow only nodes with
-mutual trust of HTTPS certificates to join the tuple-space.
-
-Weaknesses
-
-* (Currently) bootstraps via multicast discovery.
-
-* Brittle to timing, especially on virtualized clusters where clocks are unpredictable.
-
-It proved good for workload sharing -tasks can be published to it, any
-agent can say "I'm working on it" and take up the work. If the process
-fails, the task becomes available again. We used this for distributed scheduling in a rendering farm.
-
-## [Carmen](http://www.hpl.hp.com/techreports/2002/HPL-2002-257)
-
-This was another HP Labs project, related to the Cooltown "ubiquitous
-computing" work, which was a decade too early to be relevant. It was
-also positioned by management as a B2B platform, so ended up competing
-with - and losing against - WS-* and UDDI.
-
-Carmen aimed to provide service discovery with both fixed services, and
-with highly mobile client services that will roam around the network -they
-are assumed to be wireless devices.
-
-Services were published with and searched for by attributed, locality
-was considered to be a key attribute -local instances of a service
-prioritized. Those services with a static location and low rate of
-change became the stable caches of service information -becoming,
-as with skype, "supernodes".
-
-Bootstrapping the cluster relied on multicast, though alternatives
-based on DHCP and DNS were proposed.
-
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/registry/references.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/registry/references.md b/src/site/markdown/registry/references.md
deleted file mode 100644
index a5afb43..0000000
--- a/src/site/markdown/registry/references.md
+++ /dev/null
@@ -1,49 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# Service Registry References
-
-Service registration and discovery is a problem in distributed computing that has been explored for over thirty years, with
-[Birrell81]'s *Grapevine* system the first known implementation -though of
-
-# Papers
-
-* **[Birrell81]** Birrell, A. et al, [*Grapevine: An exercise in distributed computing*](http://research.microsoft.com/apps/pubs/default.aspx?id=63661). Comm. ACM 25, 4 (Apr 1982), pp260-274.
-The first documented directory service; relied on service shutdown to resolve update operations.
-
-* **[Das02]** [*SWIM: Scalable Weakly-consistent Infection-style Process Group Membership Protocol*](http://www.cs.cornell.edu/~asdas/research/dsn02-swim.pdf)
-P2P gossip-style data sharing protocol with random liveness probes to address scalable liveness checking. Ceph uses similar liveness checking.
-
-* **[Marti02]** Marti S. and Krishnam V., [*Carmen: A Dynamic Service Discovery Architecture*](http://www.hpl.hp.com/techreports/2002/HPL-2002-257),
-
-* **[Lampson86]** Lampson, B. [*Designing a Global Naming Service*](http://research.microsoft.com/en-us/um/people/blampson/36-GlobalNames/Acrobat.pdf). DEC.
-Distributed; includes an update protocol and the ability to add links to other parts of the tree. Also refers to [*Xerox Clearinghouse*](http://bitsavers.informatik.uni-stuttgart.de/pdf/xerox/parc/techReports/OPD-T8103_The_Clearinghouse.pdf), which apparently shipped.
-
-* **[Mockapetris88]** Mockapetris, P. [*Development of the domain name system*](http://bnrg.eecs.berkeley.edu/~randy/Courses/CS268.F08/papers/31_dns.pdf). The history of DNS
-
-* **[Schroeder84]** Schroeder, M.D. et al, [*Experience with Grapevine: The Growth of a Distributed System*](http://research.microsoft.com/apps/pubs/default.aspx?id=61509). Xerox.
-Writeup of the experiences of using grapevine, with its eventual consistency and lack of idempotent message delivery called out -along with coverage of operations issues.
-
-* **[van Renesse08]** van Renesse, R. et al, [*Astrolabe: A Robust and Scalable Technology For Distributed System Monitoring, Management, and Data Mining*](http://www.cs.cornell.edu/home/rvr/papers/astrolabe.pdf). ACM Transactions on Computer Systems
-Grandest P2P management framework to date; the work that earned Werner Vogel his CTO position at Amazon.
-
-* **[van Steen86]** van Steen, M. et al, [*A Scalable Location Service for Distributed Objects*](http://www.cs.vu.nl/~ast/publications/asci-1996a.pdf).
-Vrije Universiteit, Amsterdam. Probably the first Object Request Broker
-
-
-
-
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/registry/registry-model.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/registry/registry-model.md b/src/site/markdown/registry/registry-model.md
deleted file mode 100644
index 5ba2a5b..0000000
--- a/src/site/markdown/registry/registry-model.md
+++ /dev/null
@@ -1,75 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# Apache Slider: Service Registry
-
-The service registry model is designed to support dynamically
-deployed Slider applications, *and* statically deployed versions
-of the same application -provided the latter also registers itself,
-its public network services, and any configurations and files
-that it wishes clients to be able to retrieve.
-
-The architecture and implementation of this registry is not defined
-here -instead the view of it seen by clients.
-
-1. A 'service registry' exists in the YARN cluster into which
-services can be registered.
-
-1. There is no restriction on the number of services that can be registered in
-the registry, the type of service that may register, or even on how many
-registered services an application running in the YARN cluster may register.
-
-1. Services are registered by their type, owner and name. As an example,
-Alice's slider-managed HBase cluster `ingress` would have a type `org.apache.hbase`,
-owner `alice` and name `ingress`.
-
-1. In the case of Slider-managed services, there is a separate slider instance
-registration which publishes information about slider itself. In the example
-above, this would be (`org.apache.slider`,`alice`,`ingress`).
-
-1. Services can publish information about themselves, with common entries being:
-
- * service name and description.
- * URLs of published web UIs and web service APIs
- * network address of other protocols
-
-1. Services may also publish.
-
- * URLs to configuration details
- * URLs documents published for use as client-side configuration data -either
- directly or through some form of processing.
- * public service-specific data, for use by applications that are aware of
- the specific service type.
- * internal service-specific data -for use by the components that comprise
- an application. This allows the registry to be used to glue together
- the application itself.
-
-1. Services can be listed and examined.
-
-1. Service-published configuration key-value pairs can be retrieved by clients
-
-1. Service-published documents (and packages of such documents) can be
-retrieved by clients.
-
-1. There's no requirement for service instances to support any standard protocols;
-
-1. Some protocols are defined which they MAY implement. For example, the protocol
-to enumerate and retrieve configuration documents is designed to be implemented
-by any service that wishes to publish such content.
-
-1. In a secure cluster, the registry data may be restricted, along with any
-service protocols offered by the registered services.
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/registry/service_registry_end_to_end_scenario.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/registry/service_registry_end_to_end_scenario.md b/src/site/markdown/registry/service_registry_end_to_end_scenario.md
deleted file mode 100644
index 4d15fa4..0000000
--- a/src/site/markdown/registry/service_registry_end_to_end_scenario.md
+++ /dev/null
@@ -1,156 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# Apache Slider: Service Registry End-to-End Scenarios
-
-## AM startup
-
-1. AM starts, reads in configuration, creates provider
-
-2. AM builds web site, involving provider in process (*there's a possible race condition here, due to the AM registration sequence)*
-
-3. AM registers self with RM, including web and IPC ports, and receives list of existing containers; container loss notifications come in asynchronously *(which is why the AM startup process is in a synchronized block)*
-
-4. AM inits it's `ApplicationState` instance with the config, instance description and RM-supplied container list.
-
-5. AM creates service registry client using ZK quorum and path provided when AM was started
-
-6. AM registers standard endpoints: RPC, WebUI, REST APIs
-
-7. AM registers standard content it can serve (e.g `yarn-site.xml`)
-
-8. AM passes registry to provider in `bind()` operation.
-
-9. AM triggers review of application state, requesting/releasing nodes as appropriate
-
-## Agent Startup: standalone
-
-1. Container is issued to AM
-
-2. AM chooses component, launches agent on it -with URL of AM a parameter (TODO: Add registry bonding of ZK quorum and path)
-
-3. Agent starts up.
-
-4. Agent locates AM via URL/ZK info
-
-5. Agent heartbeats in with state
-
-6. AM gives agent next state command.
-
-## AM gets state from agent:
-
-1. Agent heartbeats in
-
-2. AM decides if it wants to receive config
-
-3. AM issues request for state information -all (dynamic) config data
-
-4. Agent receives it
-
-5. Agent returns all config state, including: hostnames, allocated ports, generated values (e.g. database connection strings, URLs) - as two-level (allows agent to define which config options are relevant to which document)
-
-## AM saves state for serving
-
-1. AM saves state in RAM (assumptions: small, will rebuild on restart)
-
-2. AM updates service registry with list of content that can be served up and URLs to retrieve them.
-
-3. AM fields HTTP GET requests on content
-
-## AM Serves content
-
-A simple REST service serves up content on paths published to the service registry. It is also possible to enumerate documents published by GET operations on parent paths.
-
-1. On GET command, AM locates referenced agent values
-
-2. AM builds up response document from K-V pairs. This can be in a limited set of formats: Hadoop XML, Java properties, YAML, CSV, HTTP, JSON chosen as ? type param. (this generation is done from template processing in AM using slider.core.template module)
-
-3. response is streamed with headers of : `content-type`, `content-length`, do not cache in proxy, expires,* (with expiry date chosen as ??)*
-
-# Slider Client
-
-Currently slider client enumerates the YARN registry looking for slider instances -including any instances of the same application running before launching a cluster.
-
-This
-
-* has race conditions
-* has scale limitations `O(apps-in-YARN-cluster)` + `O(completed-apps-in-RM-memory)`
-* only retrieves configuration information from slider-deployed application instances. *We do not need to restrict ourselves here.*
-
-## Slider Client lists applications
-
- slider registry --list [--servicetype <application-type>]
-
-1. Client starts
-
-2. Client creates creates service registry client using ZK quorum and path provided in client config properties (slider-client.xml)
-
-3. Client enumerates registered services and lists them
-
-## Slider Client lists content published by an application instance
-
- slider registry <instance> --listconf [--servicetype <application-type>]
-
-1. Client starts
-
-2. Client creates creates service registry client using ZK quorum and path provided in client config properties (slider-client.xml)
-
-3. Client locates registered service entry -or fails
-
-4. Client retrieves service data, specifically the listing of published documents
-
-5. Client displays list of content
-
-## Slider Client retrieves content published by an application instance
-
- slider registry <instance> --getconf <document> [--format (xml|properties|text|html|csv|yaml|json,...) [--dest <file>] [--servicetype <application-type>]
-
-1. Client starts
-
-2. Client creates creates service registry client using ZK quorum and path provided in client config properties (slider-client.xml)
-
-3. Client locates registered service entry -or fails
-
-4. Client retrieves service data, specifically the listing of published documents
-
-5. Client locates URL of content
-
-6. Client builds GET request including format
-
-7. Client executes command, follows redirects, validates content length against supplied data.
-
-8. Client prints response to console or saves to output file. This is the path specified as a destination, or, if that path refers to a directory, to
-a file underneath.
-
-## Slider Client retrieves content set published by an application instance
-
-Here a set of documents published is retrieved in the desired format of an application.
-
-## Slider Client retrieves document and applies template to it
-
-Here a set of documents published is retrieved in the desired format of an application.
-
- slider registry <instance> --source <document> [--template <path-to-template>] [--outfile <file>] [--servicetype <application-type>]
-
-1. document is retrieved as before, using a simple format such as json to retrieve it.
-
-2. The document is parsed and converted back into K-V pairs
-
-3. A template using a common/defined template library is applied to the content , generating the final output.
-
-Template paths may include local filesystem paths or (somehow) something in a package file
-
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/registry/the_YARN_application_registration_and_binding_problem.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/registry/the_YARN_application_registration_and_binding_problem.md b/src/site/markdown/registry/the_YARN_application_registration_and_binding_problem.md
deleted file mode 100644
index ab2493d..0000000
--- a/src/site/markdown/registry/the_YARN_application_registration_and_binding_problem.md
+++ /dev/null
@@ -1,192 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# Apache Slider YARN Application Registration and Binding: the Problem
-
-## March 2014
-
-# How to bind client applications to services dynamically placed applications?
-
-
-There are some constraints here
-
-1. The clients may be running outside the cluster -potentially over long-haul links.
-
-1. The location of an application deployed in a YARN cluster cannot be predicted.
-
-1. The ports used for application service endpoints cannot be hard-coded
-or predicted. (Alternatively: if they are hard-coded, then Socket-In-Use exceptions may occur)
-
-1: As components fail and get re-instantiated, their location may change.
-The rate of this depends on cluster and application stability; the longer
- lived the application, the more common it is.
-
-Existing Hadoop client apps have a configuration problem of their own:
-how are the settings in files such as `yarn-site.xml`picked up by today's
-applications? This is an issue which has historically been out of scope
-for Hadoop clusters -but if we are looking at registration and binding
-of YARN applications, there should be no reason why
-static applications cannot be discovered and bonded to using the same mechanisms.
-
-# Other constraints:
-
-1. Reduce the amount of change needed in existing applications to a minimum
----ideally none, though some pre-launch setup may be acceptable.
-
-2. Prevent malicious applications from registering a service endpoints.
-
-3. Scale with #of applications and #of clients; not overload on a cluster partitioning.
-
-4. Offer a design that works with apps that are deployed in a YARN custer
-outside of Slider. Rationale: want a mechanism that works with pure-YARN apps
-
-## Possible Solutions:
-
-### ZK
-
-Client applications use ZK to find services (addresses #1, #2 and #3).
-Requires location code in the client.
-
-HBase and Accumulo do this as part of a failover-ready design.
-
-### DNS
-
-Client apps use DNS to find services, with custom DNS server for a
-subdomain representing YARN services. Addresses #1; with a shortened TTL and
-no DNS address caching, #3. #2 addressed only if other DNS entries are used to
- publish service entries.
-
-Should support existing applications, with a configuration that is stable
-over time. It does require the clients to not cache DNS addresses forever
-(this must be explicitly set on Java applications,
-irrespective of the published TTL). It generates a load on the DNS servers
-that is `O(clients/TTL)`
-
-Google Chubby offers a DNS service to handle this. ZK does not -yet.
-
-### Floating IP Addresses
-
-If the clients know/cache IP addresses of services, these addresses could be
-floated across service instances. Linux HA has floating IP address support,
-while Docker containers can make use of them, especially if an integrated DHCP
-server handles the assignment of IP addresses to specific containers.
-
-ARP caching is the inevitable problem here, but it is still less brittle than
-relying on applications to know not to cache IP addresses -and nor does it
-place so much on DNS servers as short-TTL DNS entries.
-
-### LDAP
-
-Enterprise Directory services are used to publish/locate services. Requires
-lookup into the directory on binding (#1, #2), re-lookup on failure (#3).
-LDAP permissions can prevent untrusted applications registering.
-
-* Works well with Windows registries.
-
-* Less common Java-side, though possible -and implemented in the core Java
-libraries. Spring-LDAP is focused on connection to an LDAP server
--not LDAP-driven application config.
-
-### Registration Web Service
-
- Custom web service registration services used.
-
-* The sole WS-* one, UDDI, does not have a REST equivalent
---DNS is assumed to take on that role.
-
-* Requires new client-side code anyway.
-
-### Zookeeper URL Schema
-
-Offer our own `zk://` URL; java & .NET implementations (others?) to resolve, browser plugins.
-
-* Would address requirements #1 & #3
-
-* Cost: non-standard; needs an extension for every application/platform, and
-will not work with tools such as CURL or web browsers
-
-### AM-side config generation
-
-App-side config generation-YARN applications to generate client-side
-configuration files for launch-time information (#1, #2).
-The AM can dynamically create these, and as the storage load is all in
-the AM, does not consume as much resources in a central server as would
-publishing it all to that central server.
-
-* Requires application to know of client-side applications to support -
-and be able to generate to their configuration information (i.e. formatted files).
-
-* Requires the AM to get all information from deployed application components
-needed to generate bindings. Unless the AM can resolve YARN App templates,
-need a way to get one of the components in the app to generate settings for
-the entire cluster, and push them back.
-
-* Needs to be repeated for all YARN apps, however deployed.
-
-* Needs something similar for statically deployed applications.
-
-
-### Client-side config generation
-
-YARN app to publish attributes as key-val pairs, client-side code to read and
-generate configs from (#1, #2). Example configuration generators could
-create: Hadoop-client XML, Spring, tomcat, guice configs, something for .NET.
-
-* Not limited to Hoya application deployments only.
-
-* K-V pairs need to be published "somewhere". A structured section in the
-ZK tree per app is the obvious location -though potentially expensive. An
-alternative is AM-published data.
-
-* Needs client-side code capable of extracting information from YARN cluster
-to generate client-specific configuration.
-
-* Assumes (key, value) pairs sufficient for client config generation. Again,
-some template expansion will aid here (this time: client-side interpretation).
-
-* Client config generators need to find and bind to the target application themselves.
-
-
-
-Multiple options:
-
-* Standard ZK structure for YARN applications (maybe: YARN itself to register
-paths in ZK and set up child permissions,so enforcing security).
-
-* Agents to push to ZK dynamic information as K-V pairs
-
-* Agent provider on AM to fetch K-V pairs and include in status requests
-
-* CLI to fetch app config keys, echo out responses (needs client log4j settings
-to log all slf/log4j to stderr, so that app > results.txt would save results explicitly
-
-* client side code per app to generate specific binding information
-
-### Load-balancer app Yarn App
-
-Spread requests round a set of registered handlers, e.g web servers. Support
-plugins for session binding/sharding.
-
-Some web servers can do this already; a custom YARN app could use grizzy
-embedded. Binding problem exists, but would support scaleable dispatch of values.
-
-* Could be offered as an AM extension (in provider, ...): scales well
-with #of apps in cluster, but adds initial location/failover problems.
-
-* If offered as a core-YARN service, location is handled via a fixed
-URL. This could place high load on the service, even just 302 redirects.
-
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/release_notes/release-0.30.0.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/release_notes/release-0.30.0.md b/src/site/markdown/release_notes/release-0.30.0.md
deleted file mode 100644
index 72c6e39..0000000
--- a/src/site/markdown/release_notes/release-0.30.0.md
+++ /dev/null
@@ -1,36 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# Apache Slider Release 0.30.0 (incubating)
-
-May 2014
-
-This release is built against the Apache Hadoop 2.4.0, HBase-0.98.1
-and Accumulo 1.5.1 artifacts.
-
-Download: []()
-
-
-## Key changes
-1. Slider application registry that allow registration and discovery of application configuration and URLs (such as jmx endpoints and management UI) for client consumption.
-2. Move to a .zip packaging for Slider application packages.
-3. Richer metainfo support to provide start ordering and arbitrary template that can be published.
-4. Updated application definition packages for HBase, Accumulo, and Storm.
-
-## Other changes
-
-1. [SLIDER-13](https://issues.apache.org/jira/browse/SLIDER-13) switch build to be java7+ only.
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/security.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/security.md b/src/site/markdown/security.md
deleted file mode 100644
index 1358662..0000000
--- a/src/site/markdown/security.md
+++ /dev/null
@@ -1,197 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# Apache Slider Security
-
-This document discusses the design, implementation and use of Slider
-to deploy secure applications on a secure Hadoop cluster.
-
-### Important:
-
-This document does not cover Kerberos, how to secure a Hadoop cluster, Kerberos
-command line tools or how Hadoop uses delegation tokens to delegate permissions
-round a cluster. These are assumed, though some links to useful pages are
-listed at the bottom.
-
-
-## Concepts
-
-Slider runs in secure clusters, but with restrictions
-
-1. The keytabs to allow a worker to authenticate with the master must
- be distributed in advance: Slider does not attempt to pass these around.
-1. Until the location of Slider node instances can be strictly limited to
- a set of nodes (a future YARN feature), the keytabs must be passed to
- all the nodes in the cluster in advance, *and made available to the
- user creating the cluster*
-1. due to the way that HBase and accumulo authenticate worker nodes to
- the masters, any HBase node running on a server must authenticate as
- the same principal, and so have equal access rights to the HBase cluster.
-1. As the data directories for a slider cluster are created under the home
- directories of that user, the principals representing all role instances
- in the clusters *MUST* have read/write access to these files. This can be
- done with a shortname that matches that of the user, or by requesting
- that Slider create a directory with group write permissions -and using LDAP
- to indentify the application principals as members of the same group
- as the user.
-
-
-## Security Requirements
-
-
-### Needs
-* Slider and deployed applications to work against secure HDFS
-* Slider to work with secure YARN.
-* Slider to start secure applications
-* Kerberos and ActiveDirectory to perform the authentication.
-* Slider to only allow cluster operations by authenticated users -command line and direct RPC.
-* Any Slider Web UI and REST API for Ambari to only allow access to authenticated users.
-* The Slider database in `~/.slider/clusters/$name/data` to be writable by HBase
-
-
-### Short-lived Clusters
-* Cluster to remain secure for the duration of the Kerberos tokens issued to Slider.
-
-
-### Long-lived Clusters
-
-* Slider application instance and HBase instance to remain functional and secure over an indefinite period of time.
-
-### Initial Non-requirements
-* secure audit trail of cluster operations.
-* multiple authorized users being granted rights to a Slider Cluster (YARN admins can always kill the Slider application instance.
-* More than one HBase cluster in the YARN cluster belonging to a single user (irrespective of how they are started).
-* Any way to revoke certificates/rights of running containers.
-
-### Assumptions
-* Kerberos is running and that HDFS and YARN are running Kerberized.
-* LDAP cannot be assumed.
-* Credentials needed for HBase can be pushed out into the local filesystems of
- the of the worker nodes via some external mechanism (e.g. scp), and protected by
- the access permissions of the native filesystem. Any user with access to these
- credentials is considered to have been granted such rights.
-* These credentials can outlive the duration of the HBase containers
-* The user running HBase has the same identity as that of the HBase cluster.
-
-## Design
-
-
-1. The user is expected to have their own Kerberos principal, and have used `kinit`
- or equivalent to authenticate with Kerberos and gain a (time-bounded) TGT
-1. The user is expected to have their own principals for every host in the cluster of the form
- username/hostname@REALM
-1. A keytab must be generated which contains all these principals -and distributed
- to all the nodes in the cluster with read access permissions to the user.
-1. When the user creates a secure cluster, they provide the standard HBase kerberos options
- to identify the principals to use and the keytab location.
-
-The Slider Client will talk to HDFS and YARN authenticating itself with the TGT,
-talking to the YARN and HDFS principals which it has been configured to expect.
-
-This can be done as described in [Client Configuration] (client-configuration.html) on the command line as
-
- -D yarn.resourcemanager.principal=yarn/master@LOCAL
- -D dfs.namenode.kerberos.principal=hdfs/master@LOCAL
-
-The Slider Client will create the cluster data directory in HDFS with `rwx` permissions for
-user `r-x` for the group and `---` for others. (these can be configurable as part of the cluster options),
-
-It will then deploy the AM, which will (somehow? for how long?) retain the access
-rights of the user that created the cluster.
-
-The Application Master will read in the JSON cluster specification file, and instantiate the
-relevant number of componentss.
-
-
-## Securing communications between the Slider Client and the Slider AM.
-
-When the AM is deployed in a secure cluster,
-it automatically uses Kerberos-authorized RPC channels. The client must acquire a
-token to talk the AM.
-
-This is provided by the YARN Resource Manager when the client application
-wishes to talk with the SliderAM -a token which is only provided after
-the caller authenticates itself as the user that has access rights
-to the cluster
-
-To allow the client to freeze a Slider application instance while they are unable to acquire
-a token to authenticate with the AM, use the `--force` option.
-
-### How to enable a secure Slider client
-
-Slider can be placed into secure mode by setting the Hadoop security options:
-
-This can be done in `slider-client.xml`:
-
-
- <property>
- <name>hadoop.security.authorization</name>
- <value>true</value>
- </property>
-
- <property>
- <name>hadoop.security.authentication</name>
- <value>kerberos</value>
- </property>
-
-
-Or it can be done on the command line
-
- -D hadoop.security.authorization=true -D hadoop.security.authentication=kerberos
-
-### Adding Kerberos binding properties to the Slider Client JVM
-
-The Java Kerberos library needs to know the Kerberos controller and
-realm to use. This should happen automatically if this is set up as the
-default Kerberos binding (on a Unix system this is done in `/etc/krb5.conf`.
-
-If is not set up, a stack trace with kerberos classes at the top and
-the message `java.lang.IllegalArgumentException: Can't get Kerberos realm`
-will be printed -and the client will then fail.
-
-The realm and controller can be defined in the Java system properties
-`java.security.krb5.realm` and `java.security.krb5.kdc`. These can be fixed
-in the JVM options, as described in the [Client Configuration] (client-configuration.html)
-documentation.
-
-They can also be set on the Slider command line itself, using the `-S` parameter.
-
- -S java.security.krb5.realm=MINICLUSTER -S java.security.krb5.kdc=hadoop-kdc
-
-### Java Cryptography Exceptions
-
-
-When trying to talk to a secure, cluster you may see the message:
-
- No valid credentials provided (Mechanism level: Illegal key size)]
-
-This means that the JRE does not have the extended cryptography package
-needed to work with the keys that Kerberos needs. This must be downloaded
-from Oracle (or other supplier of the JVM) and installed according to
-its accompanying instructions.
-
-## Useful Links
-
-1. [Adding Security to Apache Hadoop](http://hortonworks.com/wp-content/uploads/2011/10/security-design_withCover-1.pdf)
-1. [The Role of Delegation Tokens in Apache Hadoop Security](http://hortonworks.com/blog/the-role-of-delegation-tokens-in-apache-hadoop-security/)
-1. [Chapter 8. Secure Apache HBase](http://hbase.apache.org/book/security.html)
-1. Hadoop Operations p135+
-1. [Java Kerberos Requirements](http://docs.oracle.com/javase/7/docs/technotes/guides/security/jgss/tutorials/KerberosReq.htmla)
-1. [Troubleshooting Kerberos on Java](http://docs.oracle.com/javase/7/docs/technotes/guides/security/jgss/tutorials/Troubleshooting.html)
-1. For OS/X users, the GUI ticket viewer is `/System/Library/CoreServices/Ticket\ Viewer.app`
-
-
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/slider_specs/application_configuration.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/slider_specs/application_configuration.md b/src/site/markdown/slider_specs/application_configuration.md
deleted file mode 100644
index 14acf2d..0000000
--- a/src/site/markdown/slider_specs/application_configuration.md
+++ /dev/null
@@ -1,82 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# Apache Slider Application Configuration
-
-App Configuration captures the default configuration associated with the application. *Details of configuration management is discussed in a separate spec*. The default configuration is modified based on user provided InstanceConfiguration, cluster specific details (e.g. HDFS root, local dir root), container allocated resources (port and hostname), and dependencies (e.g. ZK quorom hosts) and handed to the component instances.
-
-App Configuration is a folder containing all configuration needed by the application. Config files include any site.xml, log4j properties file, etc.
-
-In addition, application may have configuration parameters that do not necessarily go into a config files. Such configurations may be used during template expansion (parameters in env.sh files), as environment variables (e.g. JAVA_HOME), customize user names (for runas). These configurations can be provided as user inputs or are automatically inferred from the environment. Such configurations are stored in a file named "app_config.xml".
-
-
-
-A config file is of the form:
-
-
- <?xml version="1.0"?>
- <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
- <configuration>
- <property>
- ...
- </property>
- </configuration>
-
-
-
-Each configuration property is specified as follows:
-
-
- <property>
- <name>storm.zookeeper.session.timeout</name>
- <value>20000</value>
- <description>The session timeout for clients to ZooKeeper.</description>
- <required>false</required>
- <valueRestriction>0-30000</valueRestriction>
- </property>
- <property>
- <name>storm.zookeeper.root</name>
- <value>/storm</value>
- <description>The root location at which Storm stores data in ZK.</description>
- <required>true</required>
- </property>
- <property>
- <name>jvm.heapsize</name>
- <value>256</value>
- <description>The default JVM heap size for any component instance.</description>
- <required>true</required>
- </property>
- <property>
- <name>nimbus.host</name>
- <value>localhost</value>
- <description>The host that the master server is running on.</description>
- <required>true</required>
- <clientVisible>true</clientVisible>
- </property>
-
-
-
-* name: name of the parameter
-
-* value: default value of the parameter
-
-* description: a short description of the parameter
-
-* required: if the parameter is mandatory in which case it must have a value - default is "false"
-
-* clientVisible: if the property must be exported for a client
-