You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@slider.apache.org by bi...@apache.org on 2014/06/24 21:46:39 UTC
svn commit: r1605167 [7/9] - in /incubator/slider/site: branches/ trunk/
trunk/cgi-bin/ trunk/content/ trunk/content/css/ trunk/content/design/
trunk/content/design/registry/ trunk/content/design/specification/
trunk/content/developing/ trunk/content/d...
Added: incubator/slider/site/trunk/content/docs/manpage.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/manpage.md?rev=1605167&view=auto
==============================================================================
--- incubator/slider/site/trunk/content/docs/manpage.md (added)
+++ incubator/slider/site/trunk/content/docs/manpage.md Tue Jun 24 19:46:37 2014
@@ -0,0 +1,483 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Apache Slider: YARN-hosted applications
+
+## NAME
+
+slider -YARN-hosted applications
+
+## SYNOPSIS
+
+Slider enables applications to be dynamically created on a YARN-managed datacenter.
+The program can be used to create, pause, and shutdown the application. It can also be used to list current active
+and existing but not running "frozen" application instances.
+
+## CONCEPTS
+
+1. A *Slider application* is an application packaged to be deployed by Slider. It consists of one or more distributed *components*
+
+1. A *Slider application instance* is a slider application configured to be deployable on a specific YARN cluster, with a specific configuration. An instance can be *live* -actually running- or *frozen*. When frozen all its configuration details and instance-specific data are preserved on HDFS.
+
+1. An *image* is a *tar.gz* file containing binaries used to create the application. 1. Images are kept in the HDFS filesystem and identified by their path names; filesystem permissions can be used to share images amongst users.
+
+1. An *image configuration* is a directory that is overlaid file-by-file onto the conf/ directory inside the HBase image.
+
+1. Users can have multiple image configurations -they too are kept in HDFS, identified by their path names, and can be shared by setting the appropriate permissions, along with a configuration template file.
+
+1. Only those files provided in the image configuration directory overwrite the default values contained in the image; all other configuration files are retained.
+
+1. Late-binding properties can also be provided at create time.
+
+1. Slider can overwrite some of the configuration properties to enable the dynamically created components to bind correctly to each other.
+
+1. An *instance directory* is a directory created in HDFS describing the application instance; it records the configuration -both user specified, application-default and any dynamically created by slider.
+
+1. A user can create an application instance.
+
+1. A live instances can be *frozen*, saving its final state to its application instance state directory. All running components are shut down.
+
+1. A frozen instance can be *thawed* -a its components started on or near the servers where they were previously running.
+
+1. A frozen instance can be *destroyed*.
+
+1. Running instances can be listed.
+
+1. An instance consists of a set of components
+
+1. The supported component types depends upon the slider application.
+
+1. the count of each component must initially be specified when a application instance is created.
+
+1. Users can flex an application instance: adding or removing components dynamically.
+If the application instance is live, the changes will have immediate effect. If not, the changes will be picked up when the instance is next thawed.
+
+
+<!--- ======================================================================= -->
+
+## Invoking Slider
+
+
+ slider <ACTION> [<name>] [<OPTIONS>]
+
+
+<!--- ======================================================================= -->
+
+## COMMON COMMAND-LINE OPTIONS
+
+### `--conf configuration.xml`
+
+Configure the Slider client. This allows the filesystem, zookeeper instance and other properties to be picked up from the configuration file, rather than on the command line.
+
+Important: *this configuration file is not propagated to the application. It is purely for configuring the client itself.
+
+### `-D name=value`
+
+Define a Hadoop configuration option which overrides any options in the client configuration XML files.
+
+
+### `-m, --manager url`
+
+URL of the YARN resource manager
+
+
+### `--fs filesystem-uri`
+
+Use the specific filesystem URI as an argument to the operation.
+
+
+
+
+
+<!--- ======================================================================= -->
+
+
+<!--- ======================================================================= -->
+
+## Actions
+
+COMMANDS
+
+
+
+### `build <name>`
+
+Build an instance of the given name, with the specific options.
+
+It is not started; this can be done later with a `thaw` command.
+
+### `create <name>`
+
+Build and run an applicaton instance of the given name
+
+The `--wait` parameter, if provided, specifies the time to wait until the YARN application is actually running. Even after the YARN application has started, there may be some delay for the instance to start up.
+
+### Arguments for `build` and `create`
+
+
+
+##### `--package <uri-to-package>`
+
+This define the slider application package to be deployed.
+
+
+##### `--option <name> <value>`
+
+Set a application instance option.
+
+Example:
+
+Set an option to be passed into the `-site.xml` file of the target system, reducing
+the HDFS replication factor to 2. (
+
+ --option site.dfs.blocksize 128m
+
+Increase the number of YARN containers which must fail before the Slider application instance
+itself fails.
+
+ -O slider.container.failure.threshold 16
+
+##### `--appconf dfspath`
+
+A URI path to the configuration directory containing the template application specification. The path must be on a filesystem visible to all nodes in the YARN cluster.
+
+1. Only one configuration directory can be specified.
+1. The contents of the directory will only be read when the application instance is created/built.
+
+Example:
+
+ --appconf hdfs://namenode/users/slider/conf/hbase-template
+ --appconf file://users/accumulo/conf/template
+
+
+
+##### `--apphome localpath`
+
+A path to the home dir of a pre-installed application. If set when a Slider
+application instance is created, the instance will run with the binaries pre-installed
+on the nodes at this location
+
+*Important*: this is a path in the local filesystem which must be present
+on all hosts in the YARN cluster
+
+Example
+
+ --apphome /usr/hadoop/hbase
+
+##### `--template <filename>`
+
+Filename for the template application instance configuration. This
+will be merged with -and can overwrite- the built-in configuration options, and can
+then be overwritten by any command line `--option` and `--compopt` arguments to
+generate the final application configuration.
+
+##### `--resources <filename>`
+
+Filename for the resources configuration. This
+will be merged with -and can overwrite- the built-in resource options, and can
+then be overwritten by any command line `--resopt`, `--rescompopt` and `--component`
+arguments to generate the final resource configuration.
+
+
+##### `--image path`
+
+The full path in Hadoop HDFS to a `.tar` or `.tar.gz` file containing
+the binaries needed to run the target application.
+
+Example
+
+ --image hdfs://namenode/shared/binaries/hbase-0.96.tar.gz
+
+##### `--component <name> <count>`
+
+The desired number of instances of a component
+
+
+Example
+
+ --component worker 16
+
+This just sets the `component.instances` value of the named component's resource configuration.
+it is exactly equivalent to
+
+ --rco worker component.instances 16
+
+
+
+#### `--compopt <component> <option> <value>`
+
+Provide a specific application configuration option for a component
+
+Example
+
+ --compopt master env.TIMEOUT 10000
+
+These options are saved into the `app_conf.json` file; they are not used to configure the YARN Resource
+allocations, which must use the `--rco` parameter
+
+#### Resource Component option `--rescompopt` `--rco`
+
+`--rescompopt <component> <option> <value>`
+
+Set any role-specific option, such as its YARN memory requirements.
+
+Example
+
+ --rco worker master yarn.memory 2048
+ --rco worker worker yarn.memory max
+
+
+##### `--zkhosts host1:port1,[host2:port2,host3:port3, ...] `
+
+The zookeeper quorum.
+
+Example
+
+ --zkhosts zk1:2181,zk2:2181,zk3:4096
+
+If unset, the zookeeper quorum defined in the property `slider.zookeeper.quorum`
+is used
+
+### `destroy <name>`
+
+Destroy a (stopped) applicaton instance .
+
+Important: This deletes all persistent data
+
+Example
+
+ slider destroy instance1
+
+### `exists <name> [--live]`
+
+Probe the existence of the named Slider application instance. If the `--live` flag is set, the instance
+must actually be running
+
+If not, an error code is returned.
+
+When the --live` flag is unset, the command looks for the application instance to be
+defined in the filesystem -its operation state is not checked.
+
+Return codes
+
+ 0 : application instance is defined in the filesystem
+ 70 : application instance is unknown
+
+Example:
+
+ slider exists instance4
+
+#### Live Tests
+
+When the `--live` flag is set, the application instance must be running for the command
+to succeed
+
+1. The probe does not check the status of any Slider-deployed services, merely that a application instance has been deployed
+1. A application instance that is finished or failed is not considered to be live.
+
+Return codes
+
+ 0 : application instance is running
+ -1 : application instance exists but is not running
+ 70 : application instance is unknown
+
+
+Example:
+
+ slider exists instance4 --live
+
+### `flex <name> [--component component count]* `
+
+Flex the number of workers in an application instance to the new value. If greater than before, new copies of the component will be requested. If less, component instances will be destroyed.
+
+This operation has a return value of 0 if the size of a running instance was changed.
+
+It returns -1 if there is no running application instance, or the size of the flexed instance matches that of the original -in which case its state does not change.
+
+Example
+
+ slider flex instance1 --component worker 8 --filesystem hdfs://host:port
+ slider flex instance1 --component master 2 --filesystem hdfs://host:port
+
+
+### `freeze <name> [--force] [--wait time] [--message text]`
+
+freeze the application instance. The running application is stopped. Its settings are retained in HDFS.
+
+The `--wait` argument can specify a time in seconds to wait for the application instance to be frozen.
+
+The `--force` flag causes YARN asked directly to terminate the application instance.
+The `--message` argument supplies an optional text message to be used in
+the request: this will appear in the application's diagnostics in the YARN RM UI.
+
+If an unknown (or already frozen) application instance is named, no error is returned.
+
+Examples
+
+ slider freeze instance1 --wait 30
+ slider freeze instance2 --force --message "maintenance session"
+
+
+### `list <name>`
+
+List running Slider application instances visible to the user.
+
+If an instance name is given and there is no running instance with that name, an error is returned.
+
+Example
+
+ slider list
+ slider list instance1
+
+### `registry (--list | --listconf | --getconf <conf>) [--name <name>] [--servicetype <servicetype>] [--verbose]`
+
+List registered application instances visible to the user. This is slightly
+different from the `slider list` command in that it does not make use of the
+YARN application list. Instead it communicates with Zookeeper -and works
+with any applications which has registered itself with the "service registry"
+
+The `--name <name>` option names the registry entry to work with. For slider applications,
+this is the application instance
+
+
+The `--servicetype <servicetype>` option allows a different service type to be chosen.
+The default is `"org.apache.slider`
+
+The `--verbose` flag triggers more verbose output on the operations
+
+The `--internal` flag indicates the configurations to be listed and retrieved
+are from the "internal" list of configuration data provided for use within a
+deployed application.
+
+There are two common exit codes, the exact values being documented
+in [Exit Codes](exitcodes.html)
+
+1. If there is no matching service then the operation fails with the
+`EXIT_NOT_FOUND` code (77).
+2. If there are no configurations in a listing, or the named configuration
+is not found, the command returns the exit code `EXIT_NOT_FOUND` (77)
+
+Operations:
+
+#### `registry --list [--servicetype <servicetype>] [--name <name>] [--verbose]`
+
+List all services of the service type and optionally the name.
+
+
+
+#### `registry --listconf [--name <name>] [--internal] [--servicetype <servicetype>]`
+
+List the configurations exported by of a named application
+
+
+
+#### `registry --getconf <configuration> [--format (xml|json|properties)] [--dest <path>] [--internal] ` get the configuration
+
+Get a named configuration in a chosen format. Default: XML
+
+ `--dest <path>` : the filename or directory to save a configuration to.
+ `--format (xml|json|properties)` defines the output format
+
+
+
+
+
+
+
+### `status <name> [--out <filename>]`
+
+Get the status of the named application instance in JSON format. A filename can be used to
+specify the destination.
+
+Examples:
+
+ slider status instance1 --manager host:port
+
+ slider status instance2 --manager host:port --out status.json
+
+
+
+### `thaw <name> [--wait time`]
+
+Resume a frozen application instance, recreating it from its previously saved state. This will include a best-effort attempt to create the same number of nodes as before, though their locations may be different.
+
+Examples:
+
+ slider thaw instance2
+ slider thaw instance1 --wait 60
+
+
+If the application instance is already running, this command will not affect it.
+
+
+### `version`
+
+The command `slider version` prints out information about the compiled
+Slider application, the version of Hadoop against which it was built -and
+the version of Hadoop that is currently on its classpath.
+
+Note that this is the client-side Hadoop version, not that running on the server, though
+that can be obtained in the status operation
+
+
+
+## Commands for testing
+
+
+These are clearly abnormal operations; they are here primarily for testing
+-and documented for completeness.
+
+### `kill-container <name> --id container-id`
+
+Kill a YARN container belong to the application. This is useful primarily for testing the
+resilience to failures.
+
+Container IDs can be determined from the application instance status JSON document.
+
+
+### `am-suicide <name> [--exitcode code] [--message message] [--wait time]`
+
+This operation is purely for testing Slider Application Master restart;
+it triggers an asynchronous self-destruct operation in the AM -an
+operation that does not make any attempt to cleanly shut down the process.
+
+If the application has not exceeded its restart limit (as set by
+`slider.yarn.restart.limit`), YARN will attempt to restart the failed application.
+
+Example
+
+ slider am-suicide --exitcode 1 --wait 5000 -message "test"
+
+<!--- ======================================================================= -->
+
+
+## Instance Naming
+
+Application instance names must:
+
+1. be at least one character long
+1. begin with a lower case letter
+1. All other characters must be in the range \[a-z,0-9,_]
+1. All upper case characters are converted to lower case
+
+Example valid names:
+
+ slider1
+ storm4
+ hbase_instance
+ accumulo_m1_tserve4
+
Propchange: incubator/slider/site/trunk/content/docs/manpage.md
------------------------------------------------------------------------------
svn:eol-style = native
Added: incubator/slider/site/trunk/content/docs/security.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/security.md?rev=1605167&view=auto
==============================================================================
--- incubator/slider/site/trunk/content/docs/security.md (added)
+++ incubator/slider/site/trunk/content/docs/security.md Tue Jun 24 19:46:37 2014
@@ -0,0 +1,197 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Apache Slider Security
+
+This document discusses the design, implementation and use of Slider
+to deploy secure applications on a secure Hadoop cluster.
+
+### Important:
+
+This document does not cover Kerberos, how to secure a Hadoop cluster, Kerberos
+command line tools or how Hadoop uses delegation tokens to delegate permissions
+round a cluster. These are assumed, though some links to useful pages are
+listed at the bottom.
+
+
+## Concepts
+
+Slider runs in secure clusters, but with restrictions
+
+1. The keytabs to allow a worker to authenticate with the master must
+ be distributed in advance: Slider does not attempt to pass these around.
+1. Until the location of Slider node instances can be strictly limited to
+ a set of nodes (a future YARN feature), the keytabs must be passed to
+ all the nodes in the cluster in advance, *and made available to the
+ user creating the cluster*
+1. due to the way that HBase and accumulo authenticate worker nodes to
+ the masters, any HBase node running on a server must authenticate as
+ the same principal, and so have equal access rights to the HBase cluster.
+1. As the data directories for a slider cluster are created under the home
+ directories of that user, the principals representing all role instances
+ in the clusters *MUST* have read/write access to these files. This can be
+ done with a shortname that matches that of the user, or by requesting
+ that Slider create a directory with group write permissions -and using LDAP
+ to indentify the application principals as members of the same group
+ as the user.
+
+
+## Security Requirements
+
+
+### Needs
+* Slider and deployed applications to work against secure HDFS
+* Slider to work with secure YARN.
+* Slider to start secure applications
+* Kerberos and ActiveDirectory to perform the authentication.
+* Slider to only allow cluster operations by authenticated users -command line and direct RPC.
+* Any Slider Web UI and REST API for Ambari to only allow access to authenticated users.
+* The Slider database in `~/.slider/clusters/$name/data` to be writable by HBase
+
+
+### Short-lived Clusters
+* Cluster to remain secure for the duration of the Kerberos tokens issued to Slider.
+
+
+### Long-lived Clusters
+
+* Slider application instance and HBase instance to remain functional and secure over an indefinite period of time.
+
+### Initial Non-requirements
+* secure audit trail of cluster operations.
+* multiple authorized users being granted rights to a Slider Cluster (YARN admins can always kill the Slider application instance.
+* More than one HBase cluster in the YARN cluster belonging to a single user (irrespective of how they are started).
+* Any way to revoke certificates/rights of running containers.
+
+### Assumptions
+* Kerberos is running and that HDFS and YARN are running Kerberized.
+* LDAP cannot be assumed.
+* Credentials needed for HBase can be pushed out into the local filesystems of
+ the of the worker nodes via some external mechanism (e.g. scp), and protected by
+ the access permissions of the native filesystem. Any user with access to these
+ credentials is considered to have been granted such rights.
+* These credentials can outlive the duration of the HBase containers
+* The user running HBase has the same identity as that of the HBase cluster.
+
+## Design
+
+
+1. The user is expected to have their own Kerberos principal, and have used `kinit`
+ or equivalent to authenticate with Kerberos and gain a (time-bounded) TGT
+1. The user is expected to have their own principals for every host in the cluster of the form
+ username/hostname@REALM
+1. A keytab must be generated which contains all these principals -and distributed
+ to all the nodes in the cluster with read access permissions to the user.
+1. When the user creates a secure cluster, they provide the standard HBase kerberos options
+ to identify the principals to use and the keytab location.
+
+The Slider Client will talk to HDFS and YARN authenticating itself with the TGT,
+talking to the YARN and HDFS principals which it has been configured to expect.
+
+This can be done as described in [Client Configuration] (/docs/client-configuration.html) on the command line as
+
+ -D yarn.resourcemanager.principal=yarn/master@LOCAL
+ -D dfs.namenode.kerberos.principal=hdfs/master@LOCAL
+
+The Slider Client will create the cluster data directory in HDFS with `rwx` permissions for
+user `r-x` for the group and `---` for others. (these can be configurable as part of the cluster options),
+
+It will then deploy the AM, which will (somehow? for how long?) retain the access
+rights of the user that created the cluster.
+
+The Application Master will read in the JSON cluster specification file, and instantiate the
+relevant number of componentss.
+
+
+## Securing communications between the Slider Client and the Slider AM.
+
+When the AM is deployed in a secure cluster,
+it automatically uses Kerberos-authorized RPC channels. The client must acquire a
+token to talk the AM.
+
+This is provided by the YARN Resource Manager when the client application
+wishes to talk with the SliderAM -a token which is only provided after
+the caller authenticates itself as the user that has access rights
+to the cluster
+
+To allow the client to freeze a Slider application instance while they are unable to acquire
+a token to authenticate with the AM, use the `--force` option.
+
+### How to enable a secure Slider client
+
+Slider can be placed into secure mode by setting the Hadoop security options:
+
+This can be done in `slider-client.xml`:
+
+
+ <property>
+ <name>hadoop.security.authorization</name>
+ <value>true</value>
+ </property>
+
+ <property>
+ <name>hadoop.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+
+
+Or it can be done on the command line
+
+ -D hadoop.security.authorization=true -D hadoop.security.authentication=kerberos
+
+### Adding Kerberos binding properties to the Slider Client JVM
+
+The Java Kerberos library needs to know the Kerberos controller and
+realm to use. This should happen automatically if this is set up as the
+default Kerberos binding (on a Unix system this is done in `/etc/krb5.conf`.
+
+If is not set up, a stack trace with kerberos classes at the top and
+the message `java.lang.IllegalArgumentException: Can't get Kerberos realm`
+will be printed -and the client will then fail.
+
+The realm and controller can be defined in the Java system properties
+`java.security.krb5.realm` and `java.security.krb5.kdc`. These can be fixed
+in the JVM options, as described in the [Client Configuration] (/docs/client-configuration.html)
+documentation.
+
+They can also be set on the Slider command line itself, using the `-S` parameter.
+
+ -S java.security.krb5.realm=MINICLUSTER -S java.security.krb5.kdc=hadoop-kdc
+
+### Java Cryptography Exceptions
+
+
+When trying to talk to a secure, cluster you may see the message:
+
+ No valid credentials provided (Mechanism level: Illegal key size)]
+
+This means that the JRE does not have the extended cryptography package
+needed to work with the keys that Kerberos needs. This must be downloaded
+from Oracle (or other supplier of the JVM) and installed according to
+its accompanying instructions.
+
+## Useful Links
+
+1. [Adding Security to Apache Hadoop](http://hortonworks.com/wp-content/uploads/2011/10/security-design_withCover-1.pdf)
+1. [The Role of Delegation Tokens in Apache Hadoop Security](http://hortonworks.com/blog/the-role-of-delegation-tokens-in-apache-hadoop-security/)
+1. [Chapter 8. Secure Apache HBase](http://hbase.apache.org/book/security.html)
+1. Hadoop Operations p135+
+1. [Java Kerberos Requirements](http://docs.oracle.com/javase/7/docs/technotes/guides/security/jgss/tutorials/KerberosReq.htmla)
+1. [Troubleshooting Kerberos on Java](http://docs.oracle.com/javase/7/docs/technotes/guides/security/jgss/tutorials/Troubleshooting.html)
+1. For OS/X users, the GUI ticket viewer is `/System/Library/CoreServices/Ticket\ Viewer.app`
+
+
Propchange: incubator/slider/site/trunk/content/docs/security.md
------------------------------------------------------------------------------
svn:eol-style = native
Added: incubator/slider/site/trunk/content/docs/slider_specs/application_configuration.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/slider_specs/application_configuration.md?rev=1605167&view=auto
==============================================================================
--- incubator/slider/site/trunk/content/docs/slider_specs/application_configuration.md (added)
+++ incubator/slider/site/trunk/content/docs/slider_specs/application_configuration.md Tue Jun 24 19:46:37 2014
@@ -0,0 +1,82 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Apache Slider Application Configuration
+
+App Configuration captures the default configuration associated with the application. *Details of configuration management is discussed in a separate spec*. The default configuration is modified based on user provided InstanceConfiguration, cluster specific details (e.g. HDFS root, local dir root), container allocated resources (port and hostname), and dependencies (e.g. ZK quorom hosts) and handed to the component instances.
+
+App Configuration is a folder containing all configuration needed by the application. Config files include any site.xml, log4j properties file, etc.
+
+In addition, application may have configuration parameters that do not necessarily go into a config files. Such configurations may be used during template expansion (parameters in env.sh files), as environment variables (e.g. JAVA_HOME), customize user names (for runas). These configurations can be provided as user inputs or are automatically inferred from the environment. Such configurations are stored in a file named "app_config.xml".
+
+
+
+A config file is of the form:
+
+
+ <?xml version="1.0"?>
+ <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+ <configuration>
+ <property>
+ ...
+ </property>
+ </configuration>
+
+
+
+Each configuration property is specified as follows:
+
+
+ <property>
+ <name>storm.zookeeper.session.timeout</name>
+ <value>20000</value>
+ <description>The session timeout for clients to ZooKeeper.</description>
+ <required>false</required>
+ <valueRestriction>0-30000</valueRestriction>
+ </property>
+ <property>
+ <name>storm.zookeeper.root</name>
+ <value>/storm</value>
+ <description>The root location at which Storm stores data in ZK.</description>
+ <required>true</required>
+ </property>
+ <property>
+ <name>jvm.heapsize</name>
+ <value>256</value>
+ <description>The default JVM heap size for any component instance.</description>
+ <required>true</required>
+ </property>
+ <property>
+ <name>nimbus.host</name>
+ <value>localhost</value>
+ <description>The host that the master server is running on.</description>
+ <required>true</required>
+ <clientVisible>true</clientVisible>
+ </property>
+
+
+
+* name: name of the parameter
+
+* value: default value of the parameter
+
+* description: a short description of the parameter
+
+* required: if the parameter is mandatory in which case it must have a value - default is "false"
+
+* clientVisible: if the property must be exported for a client
+
Propchange: incubator/slider/site/trunk/content/docs/slider_specs/application_configuration.md
------------------------------------------------------------------------------
svn:eol-style = native
Added: incubator/slider/site/trunk/content/docs/slider_specs/application_definition.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/slider_specs/application_definition.md?rev=1605167&view=auto
==============================================================================
--- incubator/slider/site/trunk/content/docs/slider_specs/application_definition.md (added)
+++ incubator/slider/site/trunk/content/docs/slider_specs/application_definition.md Tue Jun 24 19:46:37 2014
@@ -0,0 +1,182 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Apache Slider Application Definition
+
+App definition is a declarative definition of a YARN application describing its content. The AppDefinition is used in conjunction with the [AppPackage](application_package.html). Application definition is an XML file and is packaged as metainfo.xml.
+
+## Structure
+
+*Non-mandatory fields are described in **italics**.*
+
+The fields to describe an application is as follows:
+
+* **name**: the name of the application
+
+* **version**: the version of the application. name and version together uniquely identify an application.
+
+* **type**: the type of the application. "YARN-APP" identifies an application definition suitable for YARN.
+
+* **minHadoopVersion**: the minimum version of hadoop on which the app can run
+
+* **components**: the list of component that the application is comprised of
+
+* **osSpecifics**: OS specific package information for the application
+
+* *commandScript*: application wide commands may also be defined. The command is executed on a component instance that is a client
+
+* *dependencies*: application can define a list of dependencies. Dependencies can be on the base services such as HDFS, ZOOKEEPER, YARN which are infrastructure services or GANGLIA, NAGIOS, etc. which are monitoring/alert services. The dependencies are parsed by the management infrastructure to provide the necessary configurations to allow the app to access the services. For example, a HDFS folder could be requested by the app to store its data, a ZOOKEEPER node to co-ordinate among components.
+
+An application contains several component. The fields associated with a component are:
+
+* **name**: name of the component
+
+* **category**: type of the component - MASTER, SLAVE, and CLIENT
+
+* **minInstanceCount**: the minimum number of instances required for this component
+
+* *maxInstanceCount*: maximum number of instances allowed for a component
+
+* **commandScript**: the script that implements the commands.
+
+ * **script**: the script location - relative to the AppPackage root
+
+ * **scriptType**: type of the script
+
+ * **timeout**: default timeout of the script
+
+* *customCommands*: any additional commands available for the component and their implementation
+
+An application definition also includes the package used to install the application. Its typically a tarball or some other form of package that does not require root access to install. The details of what happens during install is captured in the command script.
+
+* **osSpecific**: details on a per OS basis
+
+* **osType**: "any" refers to any OS ~ typical for tarballs
+
+* **packages**: list of packages that needs to be deployed
+
+* **type**: type of package
+
+* **name**: name of the package
+
+* **location**: location of the package (can be a relative folder within the parent AppPackage)
+
+Application can define a an order of activation which dictates if some component activation must follow the successful activation of other components.
+
+* **command**: specifies the component and the command in the form component-command *currently, START is the only valid command*
+
+* **requires**: specifies component and their state that the command depends on, provided in the form component-state *currently, STARTED is the only valid state*
+
+Applications can also advertise a set of properties (typically urls) that can only be bound when the application components are active. One such item can be the jmx endpoint. The properties to be advertised are organized as export groups (exportGroup) and each group can export one or more properties organized as a property bag. These values are visible through the registry service.
+
+* **name**: specifies the name of the export group
+
+Each exportGroup contains one or more exports.
+
+* **name**: the name of the export
+
+* **value**: the template that will be populated by Slider and then exported
+
+
+ <metainfo>
+ <schemaVersion>2.0</schemaVersion>
+ <application>
+ <name>HBASE</name>
+ <version>0.96.0.2.1.1</version>
+ <type>YARN-APP</type>
+ <minHadoopVersion>2.1.0</minHadoopVersion>
+ <components>
+ <component>
+ <name>HBASE_MASTER</name>
+ <category>MASTER</category>
+ <minInstanceCount>1</minInstanceCount>
+ <maxInstanceCount>2</maxInstanceCount>
+ <commandScript>
+ <script>scripts/hbase_master.py</script>
+ <scriptType>PYTHON</scriptType>
+ <timeout>600</timeout>
+ </commandScript>
+ <customCommands>
+ <customCommand>
+ <name>GRACEFUL_STOP</name>
+ <commandScript>
+ <script>scripts/hbase_master.py</script>
+ <scriptType>PYTHON</scriptType>
+ <timeout>1800</timeout>
+ </commandScript>
+ </customCommand>
+ </customCommands>
+ </component>
+
+ <component>
+ <name>HBASE_REGIONSERVER</name>
+ <category>SLAVE</category>
+ <minInstanceCount>1</minInstanceCount>
+ ...
+ </component>
+
+ <component>
+ <name>HBASE_CLIENT</name>
+ <category>CLIENT</category>
+ ...
+ </components>
+
+ <osSpecifics>
+ <osSpecific>
+ <osType>any</osType>
+ <packages>
+ <package>
+ <type>tarball</type>
+ <name>hbase-0.96.1-tar.gz</name>
+ <location>package/files</location>
+ </package>
+ </packages>
+ </osSpecific>
+ </osSpecifics>
+
+ <commandOrders>
+ <commandOrder>
+ <command>HBASE_REGIONSERVER-START</command>
+ <requires>HBASE_MASTER-STARTED</requires>
+ </commandOrder>
+ </commandOrders>
+
+ <exportGroups>
+ <exportGroup>
+ <name>QuickLinks</name>
+ <exports>
+ <export>
+ <name>JMX_Endpoint</name>
+ <value>http://${HBASE_MASTER_HOST}:${site.hbase-site.hbase.master.info.port}/jmx</value>
+ </export>
+ <export>
+ <name>Master_Status</name>
+ <value>http://${HBASE_MASTER_HOST}:${site.hbase-site.hbase.master.info.port}/master-status</value>
+ </export>
+ </exports>
+ </exportGroup>
+ </exportGroups>
+
+ <commandScript>
+ <script>scripts/app_health_check.py</script>
+ <scriptType>PYTHON</scriptType>
+ <timeout>300</timeout>
+ </commandScript>
+
+ </application>
+ </metainfo>
+
Propchange: incubator/slider/site/trunk/content/docs/slider_specs/application_definition.md
------------------------------------------------------------------------------
svn:eol-style = native
Added: incubator/slider/site/trunk/content/docs/slider_specs/application_instance_configuration.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/slider_specs/application_instance_configuration.md?rev=1605167&view=auto
==============================================================================
--- incubator/slider/site/trunk/content/docs/slider_specs/application_instance_configuration.md (added)
+++ incubator/slider/site/trunk/content/docs/slider_specs/application_instance_configuration.md Tue Jun 24 19:46:37 2014
@@ -0,0 +1,104 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Apache Slider App Instance Configuration
+
+App Instance Configuration is the configuration override provided by the application owner when creating an application instance using Slider. This configuration values override the default configuration available in the App Package.
+
+Instance configuration is a JSON formatted doc in the following form:
+
+
+ {
+ "schema": "http://example.org/specification/v2.0.0",
+ "metadata": {
+ },
+ "global": {
+ },
+ }
+
+An appConfig.json contains the application configuration. The sample below shows configuration for HBase.
+
+
+ {
+ "schema" : "http://example.org/specification/v2.0.0",
+ "metadata" : {
+ },
+ "global" : {
+ "config_types": "core-site,hdfs-site,hbase-site",
+
+ "java_home": "/usr/jdk64/jdk1.7.0_45",
+ "package_list": "files/hbase-0.96.1-hadoop2-bin.tar",
+
+ "site.global.app_user": "yarn",
+ "site.global.app_log_dir": "${AGENT_LOG_ROOT}/app/log",
+ "site.global.app_pid_dir": "${AGENT_WORK_ROOT}/app/run",
+ "site.global.security_enabled": "false",
+
+ "site.hbase-site.hbase.hstore.flush.retries.number": "120",
+ "site.hbase-site.hbase.client.keyvalue.maxsize": "10485760",
+ "site.hbase-site.hbase.hstore.compactionThreshold": "3",
+ "site.hbase-site.hbase.rootdir": "${NN_URI}/apps/hbase/data",
+ "site.hbase-site.hbase.tmp.dir": "${AGENT_WORK_ROOT}/work/app/tmp",
+ "site.hbase-site.hbase.master.info.port": "${HBASE_MASTER.ALLOCATED_PORT}",
+ "site.hbase-site.hbase.regionserver.port": "0",
+
+ "site.core-site.fs.defaultFS": "${NN_URI}",
+ "site.hdfs-site.dfs.namenode.https-address": "${NN_HOST}:50470",
+ "site.hdfs-site.dfs.namenode.http-address": "${NN_HOST}:50070"
+ }
+ }
+
+appConf.json allows you to pass in arbitrary set of configuration that Slider will forward to the application component instances.
+
+**Variable naming convention**
+In order to understand how the naming convention work, lets look at how the config is passed on to component commands. Slider agent recevies a structured bag of commands as input for all commands, INSTALL, CONFIGURE, START, etc. The command includes a section "configuration" which has config properties arranged into named property bags.
+
+* Variables of the form `site.xx.yy` translates to variables by the name `yy` within the group `xx` and are typically converted to site config files by the name `xx` containing variable `yy`. For example, `"site.hbase-site.hbase.regionserver.port":""` will be sent to the Slider-Agent as `"hbase-site" : { "hbase.regionserver.port": ""}` and app definition scripts can access all variables under `hbase-site` as a single property bag.
+* Similarly, `site.core-site.fs.defaultFS` allows you to pass in the default fs. *This specific variable is automatically made available by Slider but its shown here as an example.*
+* Variables of the form `site.global.zz` are sent in the same manner as other site variables except these variables are not expected to get translated to a site xml file. Usually, variables needed for template or other filter conditions (such as security_enabled = true/false) can be sent in as "global variable".
+
+**slider variables**
+
+* Any config not of the form `site.xx.yy` are consumed by Slider itself. Some of the manadatory configuration are:
+ * `agent.conf`: location of the agent config file (typically, "/slider/agent/conf/agent.ini")
+ * `application.def`: location of the application definition package (typically, "/slider/hbase_v096.zip")
+ * `config_types`: list of config types sent to the containers (e.g. "core-site,hdfs-site,hbase-site")
+ * `java_home`: java home path (e.g. "/usr/jdk64/jdk1.7.0_45")
+ * `package_list`: location of the package relative to the root where AppPackage is installed (e.g. "files/hbase-0.96.1-hadoop2-bin.tar.gz"
+
+**dynamically allocated ports**
+
+Apps can ask port to be dynamically assigned by Slider or they can leave it as "0". If there is a need for advertising any listening endpoint then the ports can be marked such.
+
+For example, HBase master info port needs to be advertised so that jmx endpoint can be accessed. This is indicated by using a special value, of the form, ${COMPONENT_NAME.ALLOCATED_PORT}. E.g. "site.hbase-site.hbase.master.info.port": "${HBASE_MASTER.ALLOCATED_PORT}"
+
+[Application Definition](application_definition.md) describes how to advertise arbitrary set of properties that are dynamically finalized when application is activated.
+
+**configuraing an app for ganglia metrics**
+
+There is no set guideline for doing so. How an application emits metrics and how the metrics are emitted to the right place is completely defined by the application. In the following example, we hso how HBase app is configured to emit metrics to a ganglia server.
+
+Ganglia server lifecycle is not controlled by the app instance. So the app instance only needs to know where to emit the metrics. This is achieved by three global variables
+
+* "site.global.ganglia_server_host": "gangliaserver.my.org"
+* "site.global.ganglia_server_port": "8663"
+* "site.global.ganglia_server_id": "HBaseApplicationCluster3"
+
+All three variable values are user provided. It is also expected that a gmond server is available on host gangliaserver.my.org listening for metrics at port 8663 and is named "HBaseApplicationCluster3". Its the reponsibility of the ganglia server admin to ensure that the server is unique and is only receving metrics from the application instance.
+
+
+
Propchange: incubator/slider/site/trunk/content/docs/slider_specs/application_instance_configuration.md
------------------------------------------------------------------------------
svn:eol-style = native
Added: incubator/slider/site/trunk/content/docs/slider_specs/application_needs.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/slider_specs/application_needs.md?rev=1605167&view=auto
==============================================================================
--- incubator/slider/site/trunk/content/docs/slider_specs/application_needs.md (added)
+++ incubator/slider/site/trunk/content/docs/slider_specs/application_needs.md Tue Jun 24 19:46:37 2014
@@ -0,0 +1,140 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Apache Slider's needs of an application
+
+Slider installs and runs applications in a YARN cluster -applications that
+do not need to be written for YARN.
+
+What they do need to be is deployable by Slider, which means installable by YARN,
+configurable by Slider, and, finally, executable by YARN. YARN will kill the
+executed process when destroying a container, so the deployed application
+must expect this to happen and be able to start up from a kill-initiated
+shutdown without any manual recovery process.
+
+They need to locate each other dynamically, both at startup and during execution,
+because the location of processes will be unknown at startup, and may change
+due to server and process failures.
+
+## Must
+
+* Install and run from a tarball -and be run from a user that is not root.
+
+* Be self contained or have all dependencies pre-installed.
+
+* Support dynamic discovery of nodes -such as via ZK.
+
+* Nodes to rebind themselves dynamically -so if nodes are moved, the application
+can continue
+
+* Handle kill as a normal shutdown mechanism.
+
+* Support multiple instances of the application running in the same cluster,
+ with processes from different application instances sharing
+the same servers.
+
+* Operate correctly when more than one role instance in the application is
+deployed on the same physical host. (If YARN adds anti-affinity options in
+container requests this will no longer be a requirement)
+
+* Dynamically allocate any RPC or web ports -such as supporting 0 as the number
+of the port to listen on in configuration options.
+
+* Be trusted. YARN does not run code in a sandbox.
+
+* If it talks to HDFS or other parts of Hadoop, be built against/ship with
+libaries compatible with the version of Hadoop running on the cluster.
+
+* Store persistent data in HDFS (directly or indirectly) with the exact storage location
+configurable. Specifically: not to the local filesystem, and not in a hard coded location
+such as `hdfs://app/data`. Slider creates per-instance directories for
+persistent data.
+
+* Be configurable as to where any configuration directory is (or simply relative
+to the tarball). The application must not require it to be in a hard-coded
+location such as `/etc`.
+
+* Not have a fixed location for log output -such as `/var/log/something`
+
+* Run until explicitly terminated. Slider treats an application termination
+(which triggers a container release) as a failure -and reacts to it by restarting
+the container.
+
+
+
+## MUST NOT
+
+* Require human intervention at startup or termination.
+
+## SHOULD
+
+These are the features that we'd like from a service:
+
+* Publish the actual RPC and HTTP ports in a way that can be picked up, such as via ZK
+or an admin API.
+
+* Be configurable via the standard Hadoop mechanisms: text files and XML configuration files.
+If not, custom parsers/configuration generators will be required.
+
+* Support an explicit parameter to define the configuration directory.
+
+* Take late bindings params via -D args or similar
+
+* Be possible to exec without running a complex script, so that process inheritance works everywhere, including (for testing) OS/X
+
+* Provide a way for Slider to get list of nodes in cluster and status. This will let Slider detect failed worker nodes and react to it.
+
+* FUTURE: If a graceful decommissioning is preferred, have an RPC method that a Slider provider can call to invoke this.
+
+* Be location aware from startup. Example: worker nodes to be allocated tables to serve based on which tables are
+stored locally/in-rack, rather than just randomly. This will accelerate startup time.
+
+* Support simple liveness probes (such as an HTTP GET operations).
+
+* Return a well documented set of exit codes, so that failures can be propagated
+ and understood.
+
+* Support cluster size flexing: the dynamic addition and removal of nodes.
+
+
+* Support a management platform such as Apache Ambari -so that the operational
+state of a Slider application can be monitored.
+
+## MAY
+
+* Include a single process that will run at a fixed location and whose termination
+can trigger application termination. Such a process will be executed
+in the same container as the Slider AM, and so known before all other containers
+are requested. If a live cluster is unable to handle restart/migration of
+such a process, then the Slider application will be unable to handle
+Slider AM restarts.
+
+* Ideally: report on load/cost of decommissioning.
+ E.g amount of data; app load.
+
+
+## MAY NOT
+
+* Be written for YARN.
+
+* Be (pure) Java. If the tarball contains native binaries for the cluster's hardware & OS,
+ they should be executable.
+
+* Be dynamically reconfigurable, except for the special requirement of handling
+movement of manager/peer containers in an application-specific manner.
+
+
Propchange: incubator/slider/site/trunk/content/docs/slider_specs/application_needs.md
------------------------------------------------------------------------------
svn:eol-style = native
Added: incubator/slider/site/trunk/content/docs/slider_specs/application_package.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/slider_specs/application_package.md?rev=1605167&view=auto
==============================================================================
--- incubator/slider/site/trunk/content/docs/slider_specs/application_package.md (added)
+++ incubator/slider/site/trunk/content/docs/slider_specs/application_package.md Tue Jun 24 19:46:37 2014
@@ -0,0 +1,147 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Apache Slider App Package
+
+All application artifacts, app definition, app configuration, scripts are packaged into a structured single package that can be handed off to any YARN application deployment tool including Slider
+
+## Overall Structure
+
+App package is a zip package containing all application artifacts. App package contains the following items:
+
+* **app definition file**
+application structure, content, definition, supported platforms, version, etc.
+
+* **default configurations folder**
+various configurations and configuration files associated with the application
+
+* **cmd_impl folder**
+management operations for the application/component
+
+ * **scripts folder**
+various scripts that implement management operations
+
+ * **templates folder**
+various templates used by the application
+
+ * **files folder**
+other scripts, txt files, tarballs, etc.
+
+
+
+
+The example above shows a semi-expanded view of an application "HBASE-YARN-APP" and the package structure for OOZIE command scripts.
+
+## app definition
+
+App definition is a file named "metainfo.xml". The file contains application definition as described in [Application Definition](application_definition.html).
+
+## default configurations
+
+This folder consists of various config files containing default configuration as described in [App Configuration](application_configuration.html).
+
+## package folder
+
+package includes the "implementation" of all management operations. The folders within are divided into scripts, templates, and files.
+
+### scripts folder
+
+Scripts are the implementation of management operations. There are five default operations and a composite operation. "restart" can be redefined to have a custom implementation.
+
+1. install
+
+2. configure
+
+3. start
+
+4. stop
+
+5. status
+
+6. restart (by default calls stop + start)
+
+The script specified in the metainfo is expected to understand the command. It can choose to call other scripts based on how the application author organizes the code base. For example:
+
+
+ class OozieServer(Script):
+ def install(self, env):
+ self.install_packages(env)
+
+ def configure(self, env):
+ import params
+ env.set_params(params)
+ oozie(is_server=True)
+
+ def start(self, env):
+ import params
+ env.set_params(params)
+ self.configure(env)
+ oozie_service(action='start')
+
+ def stop(self, env):
+ import params
+ env.set_params(params)
+ oozie_service(action='stop')
+
+ def status(self, env):
+ import status_params
+ env.set_params(status_params)
+ check_process_status(status_params.pid_file)
+
+
+
+The scripts are invoked in the following manner:
+
+ python SCRIPT COMMAND JSON_FILE PACKAGE_ROOT STRUCTURED_OUT_FILE
+
+* SCRIPT is the top level script that implements the commands for the component.
+
+* COMMAND is one of the six commands listed above or can be a custom command as defined in Application Definition
+
+* JSON_FILE includes all configuration parameters and the values
+
+* PACKAGE_ROOT is the root folder of the package. From this folder, its possible to access files, scripts, templates, packages (e.g. tarballs), etc. The App author has complete control over the structure of the package
+
+* STRUCTURED_OUT_FILE is the file where the script can output structured data. The management infrastructure is expected to automatically reports back STD_OUT and STD_ERR.
+
+A separate document (link TBD) discusses how the scripts are developed and the structure of the JSON_FILE containing the parameters.
+
+### templates folder
+
+templates are configurable text files that are NOT regular config files. *A library has been developed that can materialize a complete site configuration file from a property bag and therefore are not generated from templates.* Other files such as env sh files, log4j properties file, etc. may be derived from a template. Again, the implementor can choose to create these files from scratch and not use templates. The following operations are allowed during template expansion:
+
+* variable expansion
+
+* if condition
+
+* for loop
+
+* ...
+
+Sample template file for dfs.exclude file to list excluded/decommissioned hosts. hdfs_exclude_files in the property defined in params.py which is populated from config parameters defined in JSON_FILE.
+
+ {% if hdfs_exclude_file %}
+ {% for host in hdfs_exclude_file %}
+ {{host}}
+ {% endfor %}
+ {% endif %}
+
+
+### files folder
+
+files is a directory to store any other files that are needed for management operations. Sample files stored here are tarballs used to install the application, shell scripts used by various operations.
+
Propchange: incubator/slider/site/trunk/content/docs/slider_specs/application_package.md
------------------------------------------------------------------------------
svn:eol-style = native
Added: incubator/slider/site/trunk/content/docs/slider_specs/canonical_scenarios.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/slider_specs/canonical_scenarios.md?rev=1605167&view=auto
==============================================================================
--- incubator/slider/site/trunk/content/docs/slider_specs/canonical_scenarios.md (added)
+++ incubator/slider/site/trunk/content/docs/slider_specs/canonical_scenarios.md Tue Jun 24 19:46:37 2014
@@ -0,0 +1,165 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Apache Slider Guidelines for Clients and Client Applications
+
+This document will define the canonical scenarios for the deployment and management of Slider hosted applications. It will define the types of applications supported, the sequence of events for deploying the application types, and the management facilities exposed for the deployed applications.
+
+## Deployable Application Types
+
+The server-side components of an application (alternatively referred to as the application components) will be deployed and managed using a fairly uniform mechanism. However, applications can be distinguished by their associated client implementation. Specifically, each different client application type can yield different development, deployment, and management approaches. There are two primary application client types:
+
+1. **New application client** - the application deployed to Slider will interact with a newly coded application client leveraging the Slider supplied client API and management facilities. The client API will make use of distributed management facilities such as Zookeeper to provide the application client with the configuration information required for invoking remote application components. For example, the application client (or an application hosted component, e.g. a web application) will leverage the API to lookup the appropriate host(s) and port(s) for RPC based communication. Alternatively, if annotation libraries or an "app container" environment is provided, the appropriate values will be injected into the client process.
+
+2. **Existing, configurable client** - the application client predates deployment in the Slider environment, but is implemented in a manner that can be integrated with the Slider application client support facilities (APIs etc). This case is probably very similar in nature to the new application client in terms of the mechanisms it will use for component discovery, but is distinguished by the fact that itâs development pre-dates Slider. There are two possible variants of application client in this category:
+
+ 1. A client that is static - the client is dependent on existing configuration properties to communicate with master components and the existing code can not been altered (at least in the short term). This type of client would require a support infrastructure that may be relatively complex (e.g. templating of configuration files, proxying of server components).
+
+ 2. A client that can be enhanced - a client that can have its code altered can leverage a number of mechanisms (REST APIs, Zookeeper, a provided client discover API) to obtain the information required to invoke the master components of the application.
+
+## Deployment Scenarios
+
+There are two primary deployment mechanisms to examine: application component and client-side (application client) component deployment.
+
+## Application Component Deployment
+
+Applications generally are composed of one or more components. In the deployment steps below, be advised that there may be a need to repeat some of the configuration/definition steps for each component.
+
+The process of deploying applications (specifically, the non-client components of an application) is:
+
+1. Compose an application package that contains:
+
+ 1. An application definition that provides the following items:
+
+ 1. name and version of application
+
+ 2. component type(s) and role(s)
+
+ 3. system requirements (RAM, CPUs, disk space etc)
+
+ 4. ports required for RPC, UI
+
+ 5. software dependencies (HDP deployed services required in the cluster, JDK versions, python versions, etc)
+
+ 6. configurable properties including:
+
+ 1. application-specific properties. If the properties designate port numbers, the general recommendation would be to set them to zero to allow for the system to assign a dynamically selected port number that subsequently can be published to zookeeper. Other properties designating remote components running on other hosts may need some additional support (floating IPs, discovery APIs, etc). The processing of these properties by the infrastructure requires agreed upon mechanisms for identification (e.g. all properties beginning with the application name are passed through as application properties with the application name prefix stripped)
+
+ 2. Slider-specific properties. The following Slider properties are currently available:
+
+ 1. yarn.memory
+
+ 2. Environment variables specified with a "env." prefix (e.g. env.MALLOC_ARENA_MAX)
+
+ 3. role.instances
+
+ 4. role.name
+
+ 5. yarn.vcores
+
+ 2. Other application artifacts (binaries etc)
+
+2. Install the application
+
+ 1. The application package will be made available in a designated HDFS location
+
+ 2. If there is a managed application client component it will be deployed to selected nodes as defined in the cluster specification.
+
+ 3. Slider interacts with yarn (resource manager and node manager(s)) to populate the local resources on the designated nodes.
+
+ 4. Some properties important for facilitating remote interaction with the deployed components are advertised via zookeeper (though this may actually take place during component start up as the assignment of ports is a late-binding operation. Alternatively, developers may be encouraged to store these values in a registry rather than as direct-application configuration properties).
+
+## Client Application Deployment
+
+Although application clients are components that are deployed using mechanisms similar to other application components (especially in the managed case), there are a number of features that distinguish them:
+
+1. **configuration** - client applications generally require some information (host names, ports, etc) to ascertain the correct networking values required to communicate with the application's server components. In order to work in a yarn deployment, this configuration may need to be manipulated to allow proper operation (e.g. the configuration files updated with correct values, the configuration properties ascertained dynamically from services such as Zookeeper)
+
+2. **execution context **- it may be necessary to provide an execution environment for application clients that allows for discovery mechanisms (dependency injection, annotation libraries, etc)
+
+For each of these application client types there are two possible deployment modes:
+
+* **Managed** - the application client is deployed via Slider mechanisms. Clients, in this context, differ from the other application components in that they are not running, daemon processes. However, in a managed environment there is the expectation that the appropriate binaries and application elements will be distributed to the designated client hosts, and the configuration on those hosts will be updated to allow for execution of requests to the applicationâs master/server components. Therefore, client components should be defined in the application specification as elements that the management infrastructure supports (Figure 1).
+
+
+Figure 1 - Managed Application Client and associated Slider Application
+
+* **Unmanaged** - the application client is run as a process outside of Slider/yarn, although it may leverage Slider provided libraries that allow for server component discovery etc (Figure 2). These libraries would primarily be client bindings providing access to the registry leveraged by Slider (e.g. Java and python bindings to Zookeeper)
+
+
+Figure 2 - Unmanaged Application Client and associated Slider Application
+
+### Managed Application Client
+
+A managed application client is a component defined as part of the Slider/yarn application (i.e. it is part of the application definition presented to Slider). As such, it is deployed and managed via standard Slider/yarn mechanisms. This type of application client is more than likely already configured and written to work in a yarn environment.
+
+There are two primary needs to be met for a properly functioning client:
+
+1. **Discovery** - as a client, it is important that the client application retrieve the information it requires in order to communicate with the remote application components. As application components are spawned they (or the associated container agent) will advertise the relevant information using zookeeper. It will be up to the client (or the associated Slider client library) to contact zookeeper and retrieve the requested information.
+
+2. **Configuration** - there may be use cases in which a large number of configuration items are required by the client for its processing. In such cases it is more appropriate for a client to perform a bulk download of the application component(s) configuration as a JSON or XML document (via zookeeper or Slider-app comm?)
+
+Whether ascertained via discovery or bulk configuration retrieval, the attributes that the client obtains will more than likely need to be populated into the clientâs configuration files. Therefore, a templating facility or the like should be provided to allow for such configuration file manipulation.
+
+### Unmanaged Application Client
+
+An unmanaged application client is a standalone application that leverages application components deployed into the Slider/yarn environment. It is not possible to predict the deployment mechanisms or network topology employed for an unmanaged application. However, it is feasible to provide some guidance and/or software (APIs etc) to allow for application component discovery and communication.
+
+## Application Management
+
+Post deployment, the Slider infrastructure will provide the requisite set of administrative facilities required for application management, including the ability to start/stop the application, monitor the application, and reconfigure the application.
+
+### General Management
+
+There is one general management command:
+
+* List Yarn Apps - returns a listing of deployed yarn apps and associated information:
+
+ * name and version
+
+ * dependencies (required HDP services and versions, etc)
+
+ * configuration properties
+
+ * components/roles and associated configuration
+
+### Application/Component Management
+
+The following administrative functions are supported for applications:
+
+* Install the application - the installation command will take care of the population of the application resources into the pre-determined application resource directory in HDFS. The required configuration and binary directories will also be created.
+
+* start/thaw the application - Slider/Yarn runtime negotiates and instantiates the number of component containers designated by the cluster description and the components are started.
+
+* stop/freeze the application - similar to stopping, applicaiton (or a subset of their components) can be stopped.
+
+* get application status - the retrieval of application status may take a number of forms, including:
+
+ * liveness of service components
+
+ * operational metrics (application-level or component-level)
+
+ * viewing of logs
+
+* get application configuration - the configuration of application components is retrieved (JSON or XML form)
+
+* get cluster configuration - the cluster configuration is retrieved (number of various application components, associated hosts etc)
+
+* get cluster history
+
+* re-configure cluster
+
Propchange: incubator/slider/site/trunk/content/docs/slider_specs/canonical_scenarios.md
------------------------------------------------------------------------------
svn:eol-style = native
Added: incubator/slider/site/trunk/content/docs/slider_specs/creating_app_definitions.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/slider_specs/creating_app_definitions.md?rev=1605167&view=auto
==============================================================================
--- incubator/slider/site/trunk/content/docs/slider_specs/creating_app_definitions.md (added)
+++ incubator/slider/site/trunk/content/docs/slider_specs/creating_app_definitions.md Tue Jun 24 19:46:37 2014
@@ -0,0 +1,135 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Apache Slider AppPackage
+
+Slider AppPackages are a declarative definition of an application for application management. AppPackage is not a packaging scheme for application binaries and artifacts. Tarball, zip files, rpms etc. are available for that purpose. Instead AppPackage includes the application binaries along with other artifacts necessary for application management.
+
+An application instance consists of several active component such as one or more masters and several slaves. There may be a number of accompanying processes in addition to the basic master/slave processes - lets refer to all processes as app component instances. When run in the context of Yarn, the application specific processes are activated within individual Yarn Container. If you pry into an Yarn container (created through Slider) it will be apparent as to what is the role of Slider-Agent and the actual application components. The following image provides an high-level view. Within a container there are at least two processes - and instance of a slider agent and an instance of an application component. The application can itself spawn more procsses if needed.
+
+
+
+Figure 1 - High-level view of a container
+
+For example:
+
+ yarn 8849 -- python ./infra/agent/slider-agent/agent/main.py --label container_1397675825552_0011_01_000003___HBASE_REGIONSERVER --host AM_HOST --port 47830
+ yarn 9085 -- bash /hadoop/yarn/local/usercache/yarn/appcache/application_1397675825552_0011/ ... internal_start regionserver
+ yarn 9114 -- /usr/jdk64/jdk1.7.0_45/bin/java -Dproc_regionserver -XX:OnOutOfMemoryError=...
+
+The above list shows three processes, the Slider-Agent process, the bash script to start HBase Region Server and the HBase Region server itself. *Three of these together constitute the container*.
+
+## Using an AppPackage
+The following command creates an HBase application using the AppPackage for HBase.
+
+ ./slider create cl1 --image hdfs://NN:8020/slider/agent/slider-agent.tar.gz --template /work/appConf.json --resources /work/resources.json
+
+Lets analyze various parameters from the perspective of app creation:
+
+* `--image`: its the slider agent tarball
+* `--template`: app configuration
+* `--resources`: yarn resource requests
+* ⦠other parameters are described in accompanying docs.
+
+### AppPackage
+The structure of an AppPackage is described at [AppPackage](application_package.html).
+
+In the enlistment, there are three example AppPackages:
+
+* `app-packages/hbase-v0_96`
+* `app-packages/accumulo`
+* `app-packages/storm-v0_91`
+
+The above folders, with minor edits, can be packaged as *zip* files to get the corresponding AppPackages. The application tarball file, containing the binaries/artifacts of the application itself is a component within the AppPackage. They are:
+
+* For hbase - `app-packages/hbase-v0_96/package/files/hbase-0.96.1-hadoop2-bin.tar.gz.REPLACE`
+* For accumulo - `app-packages/accumulo/package/files/` (no file present)
+* For storm - `app-packages/storm-v0_91/package/files/apache-storm-0.9.1.2.1.1.0-237.tar.gz.placeholder`
+
+**They are placehoder files**, mostly because the files themselves are too large as well as users are free to use their own version of the package. To create a Slider AppPackage - replace the file with an actual application tarball and then ensure that the metainfo.xml has the correct file name. After that create a zip file using standard zip commands and ensure that the package has the metainfo.xml file at the root folder.
+
+For Accumulo, this is automated in maven by running
+
+ mvn clean package -DskipTests -Paccumulo-app-package
+
+Additional configuration parameters are described in `app-packages/accumulo/README.txt`.
+
+For other app packages, follow this example:
+
+* cd slider/app-packages/hbase
+* zip -r hbase-v098.zip .
+* Looking at the content through unzip -l "$@" hbase-v098.zip
+
+```
+Archive: hbase-v098.zip
+ Length Date Time Name
+--------- ---------- ----- ----
+ 3158 06-19-2014 22:08 appConfig.json
+ 0 06-19-2014 22:15 configuration/
+ 5077 06-19-2014 22:15 configuration/global.xml
+ 5248 06-19-2014 22:15 configuration/hbase-log4j.xml
+ 2250 06-19-2014 22:15 configuration/hbase-policy.xml
+ 14705 06-19-2014 22:15 configuration/hbase-site.xml
+ 2142 06-19-2014 22:17 jmx_metrics.json
+ 3991 06-19-2014 22:45 metainfo.xml
+ 0 06-19-2014 22:33 package/
+ 0 06-19-2014 22:31 package/files/
+ 83278591 06-19-2014 22:31 package/files/hbase-0.98.3-hadoop2-bin.tar.gz
+ 0 06-19-2014 22:29 package/scripts/
+ 787 06-19-2014 22:27 package/scripts/__init__.py
+ 1378 06-19-2014 22:27 package/scripts/functions.py
+ 3738 06-19-2014 22:27 package/scripts/hbase.py
+ 1205 06-19-2014 22:27 package/scripts/hbase_client.py
+ 1640 06-19-2014 22:27 package/scripts/hbase_master.py
+ 1764 06-19-2014 22:27 package/scripts/hbase_regionserver.py
+ 1482 06-19-2014 22:27 package/scripts/hbase_service.py
+ 5089 06-19-2014 22:27 package/scripts/params.py
+ 973 06-19-2014 22:27 package/scripts/status_params.py
+ 0 06-19-2014 22:34 package/templates/
+ 2799 06-19-2014 22:34 package/templates/hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2
+ 2799 06-19-2014 22:34 package/templates/hadoop-metrics2-hbase.properties-GANGLIA-RS.j2
+ 3878 06-19-2014 22:34 package/templates/hbase-env.sh.j2
+ 909 06-19-2014 22:34 package/templates/hbase_client_jaas.conf.j2
+ 989 06-19-2014 22:34 package/templates/hbase_master_jaas.conf.j2
+ 1001 06-19-2014 22:34 package/templates/hbase_regionserver_jaas.conf.j2
+ 837 06-19-2014 22:34 package/templates/regionservers.j2
+ 3486 06-19-2014 22:39 pom.xml
+ 1921 06-19-2014 21:53 README.txt
+ 358 06-19-2014 22:36 resources.json
+ 0 06-19-2014 22:40 src/
+ 0 06-19-2014 23:02 src/assembly/
+ 2400 06-19-2014 22:45 src/assembly/hbase.xml
+--------- -------
+ 83354595 35 files
+```
+
+Sample **resources.json** and **appConfig.json** files are also included in the enlistment. These are samples and are typically tested on one node test installations.
+
+
+### --template appConfig.json
+An appConfig.json contains the application configuration. See [Specifications InstanceConfiguration](application_instance_configuration.html) for details on how to create a template config file. The enlistment includes sample config files for HBase, Accumulo, and Storm.
+
+
+### --resources resources.json
+Resource specification is an input to Slider to specify the Yarn resource needs for each component type that belong to the application. [Specification of Resources](resource_specification.html) describes how to write a resource config json file. The enlistment includes sample config files for HBase, Accumulo, and Storm.
+
+
+## Scripting for AppPackage
+Refer to [App Command Scripts](writing_app_command_scripts) for details on how to write scripts for a AppPackage. These scripts are in the package/script folder within the AppPackage. *Use the checked in samples for HBase/Storm/Accumulo as reference for script development.*
+
+
+
Propchange: incubator/slider/site/trunk/content/docs/slider_specs/creating_app_definitions.md
------------------------------------------------------------------------------
svn:eol-style = native
Added: incubator/slider/site/trunk/content/docs/slider_specs/index.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/slider_specs/index.md?rev=1605167&view=auto
==============================================================================
--- incubator/slider/site/trunk/content/docs/slider_specs/index.md (added)
+++ incubator/slider/site/trunk/content/docs/slider_specs/index.md Tue Jun 24 19:46:37 2014
@@ -0,0 +1,53 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Apache Slider App Packaging
+
+##Introduction
+
+**SLIDER: A collection of tools and technologies to simplify the packaging, deployment and management of long-running applications on YARN.**
+
+- Availability (always-on) - YARN works with the application to ensure recovery or restart of running application components.
+- Flexibility (dynamic scaling) - YARN provides the application with the facilities to allow for scale-up or scale-down
+- Resource Mgmt (optimization) - YARN handles allocation of cluster resources.
+
+## Terminology
+
+### Apps on YARN
+ - Application written to run directly on YARN
+ - Packaging, deployment and lifecycle management are custom built for each application
+
+### Slider Apps
+ - Applications deployed and managed on YARN using Slider
+ - Use of slider minimizes custom code for deployment + lifecycle management
+ - Requires apps to follow Slider guidelines and packaging ("Sliderize")
+
+Specifications
+---
+
+The entry points to leverage Slider are:
+
+- [Application Needs](application_needs.html) What it takes to be deployable by Slider.
+- [Slider AppPackage](creating_app_definitions.html) Overview of how to create an Slider AppPackage.
+- [Specifications for AppPackage](application_package.html) Describes the structure of an AppPackage
+- [Specifications for Application Definition](application_definition.html) How to write metainfo.xml?
+- [Specifications for Configuration](application_configuration.html) How to create a template config file for an app?
+- [Specification of Resources](resource_specification.html) How to write a resource spec for an app?
+- [Specifications InstanceConfiguration](application_instance_configuration.html) How to write a template config for an app?
+- [Guidelines for Clients and Client Applications](canonical_scenarios.html)
+- [Documentation for "General Developer Guidelines"](/developing/index.html)
+
Propchange: incubator/slider/site/trunk/content/docs/slider_specs/index.md
------------------------------------------------------------------------------
svn:eol-style = native
Added: incubator/slider/site/trunk/content/docs/slider_specs/resource_specification.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/slider_specs/resource_specification.md?rev=1605167&view=auto
==============================================================================
--- incubator/slider/site/trunk/content/docs/slider_specs/resource_specification.md (added)
+++ incubator/slider/site/trunk/content/docs/slider_specs/resource_specification.md Tue Jun 24 19:46:37 2014
@@ -0,0 +1,53 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Apache Slider Resource Specification
+Resource specification is an input to Slider to specify the Yarn resource needs for each component type that belong to the application.
+
+An example resource requirement for an application that has two components "master" and "worker" is as follows. Slider will automatically add the requirements for the AppMaster for the application. This compoent is named "slider-appmaster".
+
+Some parameters that can be specified for a component instance include:
+
+* `yarn.memory`: amount of memory requried for the component instance
+* `yarn.vcores`: number of vcores requested
+* `yarn.role.priority`: each component must be assigned unique priority. Component with higher priority come up earlier than components with lower priority
+* `yarn.component.instances`: number of instances for this component type
+
+Sample:
+
+ {
+ "schema" : "http://example.org/specification/v2.0.0",
+ "metadata" : {
+ },
+ "global" : {
+ },
+ "components" : {
+ "HBASE_MASTER" : {
+ "yarn.role.priority" : "1",
+ "yarn.component.instances" : "1"
+ "yarn.memory" : "768",
+ "yarn.vcores" : "1"
+ },
+ "slider-appmaster" : {
+ },
+ "HBASE_REGIONSERVER" : {
+ "yarn.role.priority" : "2",
+ "yarn.component.instances" : "1"
+ }
+ }
+ }
+
Propchange: incubator/slider/site/trunk/content/docs/slider_specs/resource_specification.md
------------------------------------------------------------------------------
svn:eol-style = native