You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@slider.apache.org by bi...@apache.org on 2014/05/30 18:47:16 UTC
svn commit: r1598665 [13/16] - in /incubator/slider/site/content: ./
architecture/ configuration/ css/ developing/ images/ images/logos/
images/profiles/ img/ js/ registry/ release_notes/ slider_specs/
specification/
Added: incubator/slider/site/content/slider_specs/application_definition.html
URL: http://svn.apache.org/viewvc/incubator/slider/site/content/slider_specs/application_definition.html?rev=1598665&view=auto
==============================================================================
--- incubator/slider/site/content/slider_specs/application_definition.html (added)
+++ incubator/slider/site/content/slider_specs/application_definition.html Fri May 30 16:47:13 2014
@@ -0,0 +1,391 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2014-05-30
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20140530" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache Slider 0.30 (incubating) - </title>
+ <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../css/site.css" />
+ <link rel="stylesheet" href="../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+ </head>
+ <body class="topBarEnabled">
+
+
+
+
+
+ <div id="topbar" class="navbar navbar-fixed-top ">
+ <div class="navbar-inner">
+ <div class="container"><div class="nav-collapse">
+
+
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Documentation <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li class="dropdown-submenu">
+ <a href="../project-info.html" title="Project Information">Project Information</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../index.html" title="About">About</a>
+</li>
+ <li> <a href="../plugin-management.html" title="Plugin Management">Plugin Management</a>
+</li>
+ <li> <a href="../distribution-management.html" title="Distribution Management">Distribution Management</a>
+</li>
+ <li> <a href="../dependency-info.html" title="Dependency Information">Dependency Information</a>
+</li>
+ <li> <a href="../dependency-convergence.html" title="Dependency Convergence">Dependency Convergence</a>
+</li>
+ <li> <a href="../source-repository.html" title="Source Repository">Source Repository</a>
+</li>
+ <li> <a href="../mail-lists.html" title="Mailing Lists">Mailing Lists</a>
+</li>
+ <li> <a href="../issue-tracking.html" title="Issue Tracking">Issue Tracking</a>
+</li>
+ <li> <a href="../integration.html" title="Continuous Integration">Continuous Integration</a>
+</li>
+ <li> <a href="../plugins.html" title="Project Plugins">Project Plugins</a>
+</li>
+ <li> <a href="../license.html" title="Project License">Project License</a>
+</li>
+ <li> <a href="../modules.html" title="Project Modules">Project Modules</a>
+</li>
+ <li> <a href="../dependency-management.html" title="Dependency Management">Dependency Management</a>
+</li>
+ <li> <a href="../team-list.html" title="Project Team">Project Team</a>
+</li>
+ <li> <a href="../project-summary.html" title="Project Summary">Project Summary</a>
+</li>
+ <li> <a href="../dependencies.html" title="Dependencies">Dependencies</a>
+</li>
+ </ul>
+ </li>
+
+ <li class="dropdown-submenu">
+ <a href="../project-reports.html" title="Project Reports">Project Reports</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../surefire-report.html" title="Surefire Report">Surefire Report</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documents <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../getting_started.html" title="Getting Started">Getting Started</a>
+</li>
+
+ <li> <a href="../manpage.html" title="manpage">manpage</a>
+</li>
+
+ <li> <a href="../troubleshooting.html" title="Troubleshooting">Troubleshooting</a>
+</li>
+
+ <li> <a href="../architecture/index.html" title="Architecture">Architecture</a>
+</li>
+
+ <li> <a href="../developing/index.html" title="Developing">Developing</a>
+</li>
+
+ <li> <a href="../exitcodes.html" title="Exitcodes">Exitcodes</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+ <div class="container">
+ <div id="banner">
+ <div class="pull-left">
+ <a href=".././" id="bannerLeft">
+ <h2>Apache Slider (incubating)</h2>
+ </a>
+ </div>
+ <div class="pull-right"> <div id="bannerRight">
+ <img src="http://incubator.apache.org/images/apache-incubator-logo.png" />
+ </div>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li id="publishDate">Last Published: 2014-05-30</li>
+
+
+
+ <li id="projectVersion" class="pull-right">Version: 0.30</li>
+
+ </ul>
+ </div>
+
+
+
+ <div id="bodyColumn" >
+
+ <!-- -
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. --><h1>Apache Slider Application Definition</h1>
+<p>App definition is a declarative definition of a YARN application describing its content. The AppDefinition is used in conjunction with the <a href="application_package.md">AppPackage</a>. Application definition is an XML file and is packaged as metainfo.xml.</p>
+<div class="section">
+<h2>Structure<a name="Structure"></a></h2>
+<p><i>Non-mandatory fields are described in </i><i>italics</i><i>.</i></p>
+<p>The fields to describe an application is as follows:</p>
+
+<ul>
+
+<li>
+<p><b>name</b>: the name of the application</p></li>
+
+<li>
+<p><b>version</b>: the version of the application. name and version together uniquely identify an application.</p></li>
+
+<li>
+<p><b>type</b>: the type of the application. “YARN-APP” identifies an application definition suitable for YARN.</p></li>
+
+<li>
+<p><b>minHadoopVersion</b>: the minimum version of hadoop on which the app can run</p></li>
+
+<li>
+<p><b>components</b>: the list of component that the application is comprised of</p></li>
+
+<li>
+<p><b>osSpecifics</b>: OS specific package information for the application</p></li>
+
+<li>
+<p><i>commandScript</i>: application wide commands may also be defined. The command is executed on a component instance that is a client</p></li>
+
+<li>
+<p><i>dependencies</i>: application can define a list of dependencies. Dependencies can be on the base services such as HDFS, ZOOKEEPER, YARN which are infrastructure services or GANGLIA, NAGIOS, etc. which are monitoring/alert services. The dependencies are parsed by the management infrastructure to provide the necessary configurations to allow the app to access the services. For example, a HDFS folder could be requested by the app to store its data, a ZOOKEEPER node to co-ordinate among components.</p></li>
+</ul>
+<p>An application contains several component. The fields associated with a component are:</p>
+
+<ul>
+
+<li>
+<p><b>name</b>: name of the component</p></li>
+
+<li>
+<p><b>category</b>: type of the component - MASTER, SLAVE, and CLIENT</p></li>
+
+<li>
+<p><b>minInstanceCount</b>: the minimum number of instances required for this component</p></li>
+
+<li>
+<p><i>maxInstanceCount</i>: maximum number of instances allowed for a component</p></li>
+
+<li>
+<p><b>commandScript</b>: the script that implements the commands.</p></li>
+
+<li>
+<p><b>script</b>: the script location - relative to the AppPackage root</p></li>
+
+<li>
+<p><b>scriptType</b>: type of the script</p></li>
+
+<li>
+<p><b>timeout</b>: default timeout of the script</p></li>
+
+<li>
+<p><i>customCommands</i>: any additional commands available for the component and their implementation</p></li>
+</ul>
+<p>An application definition also includes the package used to install the application. Its typically a tarball or some other form of package that does not require root access to install. The details of what happens during install is captured in the command script.</p>
+
+<ul>
+
+<li>
+<p><b>osSpecific</b>: details on a per OS basis</p></li>
+
+<li>
+<p><b>osType</b>: “any” refers to any OS ~ typical for tarballs</p></li>
+
+<li>
+<p><b>packages</b>: list of packages that needs to be deployed</p></li>
+
+<li>
+<p><b>type</b>: type of package</p></li>
+
+<li>
+<p><b>name</b>: name of the package</p></li>
+
+<li>
+<p><b>location</b>: location of the package (can be a relative folder within the parent AppPackage)</p></li>
+</ul>
+<p>Application can define a an order of activation which dictates if some component activation must follow the successful activation of other components.</p>
+
+<ul>
+
+<li>
+<p><b>command</b>: specifies the component and the command in the form component-command <i>currently, START is the only valid command</i></p></li>
+
+<li>
+<p><b>requires</b>: specifies component and their state that the command depends on, provided in the form component-state <i>currently, STARTED is the only valid state</i></p></li>
+</ul>
+<p>Applications can also advertise a set of properties (typically urls) that can only be bound when the application components are active. One such item can be the jmx endpoint. The properties to be advertised are organized as export groups (exportGroup) and each group can export one or more properties organized as a property bag. These values are visible through the registry service.</p>
+
+<ul>
+
+<li><b>name</b>: specifies the name of the export group</li>
+</ul>
+<p>Each exportGroup contains one or more exports.</p>
+
+<ul>
+
+<li>
+<p><b>name</b>: the name of the export</p></li>
+
+<li>
+<p><b>value</b>: the template that will be populated by Slider and then exported</p></li>
+</ul>
+
+<div class="source">
+<pre> <metainfo>
+ <schemaVersion>2.0</schemaVersion>
+ <application>
+ <name>HBASE</name>
+ <version>0.96.0.2.1.1</version>
+ <type>YARN-APP</type>
+ <minHadoopVersion>2.1.0</minHadoopVersion>
+ <components>
+ <component>
+ <name>HBASE_MASTER</name>
+ <category>MASTER</category>
+ <minInstanceCount>1</minInstanceCount>
+ <maxInstanceCount>2</maxInstanceCount>
+ <commandScript>
+ <script>scripts/hbase_master.py</script>
+ <scriptType>PYTHON</scriptType>
+ <timeout>600</timeout>
+ </commandScript>
+ <customCommands>
+ <customCommand>
+ <name>GRACEFUL_STOP</name>
+ <commandScript>
+ <script>scripts/hbase_master.py</script>
+ <scriptType>PYTHON</scriptType>
+ <timeout>1800</timeout>
+ </commandScript>
+ </customCommand>
+ </customCommands>
+ </component>
+
+ <component>
+ <name>HBASE_REGIONSERVER</name>
+ <category>SLAVE</category>
+ <minInstanceCount>1</minInstanceCount>
+ ...
+ </component>
+
+ <component>
+ <name>HBASE_CLIENT</name>
+ <category>CLIENT</category>
+ ...
+ </components>
+
+ <osSpecifics>
+ <osSpecific>
+ <osType>any</osType>
+ <packages>
+ <package>
+ <type>tarball</type>
+ <name>hbase-0.96.1-tar.gz</name>
+ <location>package/files</location>
+ </package>
+ </packages>
+ </osSpecific>
+ </osSpecifics>
+
+ <commandOrders>
+ <commandOrder>
+ <command>HBASE_REGIONSERVER-START</command>
+ <requires>HBASE_MASTER-STARTED</requires>
+ </commandOrder>
+ </commandOrders>
+
+ <exportGroups>
+ <exportGroup>
+ <name>QuickLinks</name>
+ <exports>
+ <export>
+ <name>JMX_Endpoint</name>
+ <value>http://${HBASE_MASTER_HOST}:${site.hbase-site.hbase.master.info.port}/jmx</value>
+ </export>
+ <export>
+ <name>Master_Status</name>
+ <value>http://${HBASE_MASTER_HOST}:${site.hbase-site.hbase.master.info.port}/master-status</value>
+ </export>
+ </exports>
+ </exportGroup>
+ </exportGroups>
+
+ <commandScript>
+ <script>scripts/app_health_check.py</script>
+ <scriptType>PYTHON</scriptType>
+ <timeout>300</timeout>
+ </commandScript>
+
+ </application>
+ </metainfo>
+</pre></div></div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container">
+ <div class="row span12">Copyright © 2014
+ <a href="http://www.apache.org/">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+ <?xml version="1.0" encoding="UTF-8"?>
+<div class="row-fluid">Apache Slider, Slider, Apache, and the Apache Incubator logo are trademarks of The Apache Software Foundation.</div>
+
+ <p id="poweredBy" class="pull-right">
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../images/logos/maven-feather.png" />
+ </a>
+ </p>
+
+ </div>
+ </footer>
+ </body>
+</html>
Propchange: incubator/slider/site/content/slider_specs/application_definition.html
------------------------------------------------------------------------------
svn:eol-style = native
Added: incubator/slider/site/content/slider_specs/application_instance_configuration.html
URL: http://svn.apache.org/viewvc/incubator/slider/site/content/slider_specs/application_instance_configuration.html?rev=1598665&view=auto
==============================================================================
--- incubator/slider/site/content/slider_specs/application_instance_configuration.html (added)
+++ incubator/slider/site/content/slider_specs/application_instance_configuration.html Fri May 30 16:47:13 2014
@@ -0,0 +1,278 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2014-05-30
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20140530" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache Slider 0.30 (incubating) - </title>
+ <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../css/site.css" />
+ <link rel="stylesheet" href="../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+ </head>
+ <body class="topBarEnabled">
+
+
+
+
+
+ <div id="topbar" class="navbar navbar-fixed-top ">
+ <div class="navbar-inner">
+ <div class="container"><div class="nav-collapse">
+
+
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Documentation <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li class="dropdown-submenu">
+ <a href="../project-info.html" title="Project Information">Project Information</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../index.html" title="About">About</a>
+</li>
+ <li> <a href="../plugin-management.html" title="Plugin Management">Plugin Management</a>
+</li>
+ <li> <a href="../distribution-management.html" title="Distribution Management">Distribution Management</a>
+</li>
+ <li> <a href="../dependency-info.html" title="Dependency Information">Dependency Information</a>
+</li>
+ <li> <a href="../dependency-convergence.html" title="Dependency Convergence">Dependency Convergence</a>
+</li>
+ <li> <a href="../source-repository.html" title="Source Repository">Source Repository</a>
+</li>
+ <li> <a href="../mail-lists.html" title="Mailing Lists">Mailing Lists</a>
+</li>
+ <li> <a href="../issue-tracking.html" title="Issue Tracking">Issue Tracking</a>
+</li>
+ <li> <a href="../integration.html" title="Continuous Integration">Continuous Integration</a>
+</li>
+ <li> <a href="../plugins.html" title="Project Plugins">Project Plugins</a>
+</li>
+ <li> <a href="../license.html" title="Project License">Project License</a>
+</li>
+ <li> <a href="../modules.html" title="Project Modules">Project Modules</a>
+</li>
+ <li> <a href="../dependency-management.html" title="Dependency Management">Dependency Management</a>
+</li>
+ <li> <a href="../team-list.html" title="Project Team">Project Team</a>
+</li>
+ <li> <a href="../project-summary.html" title="Project Summary">Project Summary</a>
+</li>
+ <li> <a href="../dependencies.html" title="Dependencies">Dependencies</a>
+</li>
+ </ul>
+ </li>
+
+ <li class="dropdown-submenu">
+ <a href="../project-reports.html" title="Project Reports">Project Reports</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../surefire-report.html" title="Surefire Report">Surefire Report</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documents <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../getting_started.html" title="Getting Started">Getting Started</a>
+</li>
+
+ <li> <a href="../manpage.html" title="manpage">manpage</a>
+</li>
+
+ <li> <a href="../troubleshooting.html" title="Troubleshooting">Troubleshooting</a>
+</li>
+
+ <li> <a href="../architecture/index.html" title="Architecture">Architecture</a>
+</li>
+
+ <li> <a href="../developing/index.html" title="Developing">Developing</a>
+</li>
+
+ <li> <a href="../exitcodes.html" title="Exitcodes">Exitcodes</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+ <div class="container">
+ <div id="banner">
+ <div class="pull-left">
+ <a href=".././" id="bannerLeft">
+ <h2>Apache Slider (incubating)</h2>
+ </a>
+ </div>
+ <div class="pull-right"> <div id="bannerRight">
+ <img src="http://incubator.apache.org/images/apache-incubator-logo.png" />
+ </div>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li id="publishDate">Last Published: 2014-05-30</li>
+
+
+
+ <li id="projectVersion" class="pull-right">Version: 0.30</li>
+
+ </ul>
+ </div>
+
+
+
+ <div id="bodyColumn" >
+
+ <!-- -
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. --><h1>Apache Slider App Instance Configuration</h1>
+<p>App Instance Configuration is the configuration override provided by the application owner when creating an application instance using Slider. This configuration values override the default configuration available in the App Package.</p>
+<p>Instance configuration is a JSON formatted doc in the following form:</p>
+
+<div class="source">
+<pre>{
+ "schema": "http://example.org/specification/v2.0.0",
+ "metadata": {
+ },
+ "global": {
+ },
+}
+</pre></div>
+<p>An appConfig.json contains the application configuration. The sample below shows configuration for HBase.</p>
+
+<div class="source">
+<pre>{
+ "schema" : "http://example.org/specification/v2.0.0",
+ "metadata" : {
+ },
+ "global" : {
+ "config_types": "core-site,hdfs-site,hbase-site",
+
+ "java_home": "/usr/jdk64/jdk1.7.0_45",
+ "package_list": "files/hbase-0.96.1-hadoop2-bin.tar",
+
+ "site.global.app_user": "yarn",
+ "site.global.app_log_dir": "${AGENT_LOG_ROOT}/app/log",
+ "site.global.app_pid_dir": "${AGENT_WORK_ROOT}/app/run",
+ "site.global.security_enabled": "false",
+
+ "site.hbase-site.hbase.hstore.flush.retries.number": "120",
+ "site.hbase-site.hbase.client.keyvalue.maxsize": "10485760",
+ "site.hbase-site.hbase.hstore.compactionThreshold": "3",
+ "site.hbase-site.hbase.rootdir": "${NN_URI}/apps/hbase/data",
+ "site.hbase-site.hbase.tmp.dir": "${AGENT_WORK_ROOT}/work/app/tmp",
+ "site.hbase-site.hbase.master.info.port": "${HBASE_MASTER.ALLOCATED_PORT}",
+ "site.hbase-site.hbase.regionserver.port": "0",
+
+ "site.core-site.fs.defaultFS": "${NN_URI}",
+ "site.hdfs-site.dfs.namenode.https-address": "${NN_HOST}:50470",
+ "site.hdfs-site.dfs.namenode.http-address": "${NN_HOST}:50070"
+ }
+}
+</pre></div>
+<p>appConf.json allows you to pass in arbitrary set of configuration that Slider will forward to the application component instances.</p>
+<p><b>Variable naming convention</b> In order to understand how the naming convention work, lets look at how the config is passed on to component commands. Slider agent recevies a structured bag of commands as input for all commands, INSTALL, CONFIGURE, START, etc. The command includes a section “configuration” which has config properties arranged into named property bags.</p>
+
+<ul>
+
+<li>Variables of the form <tt>site.xx.yy</tt> translates to variables by the name <tt>yy</tt> within the group <tt>xx</tt> and are typically converted to site config files by the name <tt>xx</tt> containing variable <tt>yy</tt>. For example, <tt>"site.hbase-site.hbase.regionserver.port":""</tt> will be sent to the Slider-Agent as <tt>"hbase-site" : { "hbase.regionserver.port": ""}</tt> and app definition scripts can access all variables under <tt>hbase-site</tt> as a single property bag.</li>
+
+<li>Similarly, <tt>site.core-site.fs.defaultFS</tt> allows you to pass in the default fs. <i>This specific variable is automatically made available by Slider but its shown here as an example.</i></li>
+
+<li>Variables of the form <tt>site.global.zz</tt> are sent in the same manner as other site variables except these variables are not expected to get translated to a site xml file. Usually, variables needed for template or other filter conditions (such as security_enabled = true/false) can be sent in as “global variable”.</li>
+</ul>
+<p><b>slider variables</b></p>
+
+<ul>
+
+<li>Any config not of the form <tt>site.xx.yy</tt> are consumed by Slider itself. Some of the manadatory configuration are:</li>
+
+<li><tt>agent.conf</tt>: location of the agent config file (typically, “/slider/agent/conf/agent.ini”)</li>
+
+<li><tt>application.def</tt>: location of the application definition package (typically, “/slider/hbase_v096.zip”)</li>
+
+<li><tt>config_types</tt>: list of config types sent to the containers (e.g. “core-site,hdfs-site,hbase-site”)</li>
+
+<li><tt>java_home</tt>: java home path (e.g. “/usr/jdk64/jdk1.7.0_45”)</li>
+
+<li><tt>package_list</tt>: location of the package relative to the root where AppPackage is installed (e.g. “files/hbase-0.96.1-hadoop2-bin.tar.gz”</li>
+</ul>
+<p><b>dynamically allocated ports</b></p>
+<p>Apps can ask port to be dynamically assigned by Slider or they can leave it as “0”. If there is a need for advertising any listening endpoint then the ports can be marked such.</p>
+<p>For example, HBase master info port needs to be advertised so that jmx endpoint can be accessed. This is indicated by using a special value, of the form, ${COMPONENT_NAME.ALLOCATED_PORT}. E.g. “site.hbase-site.hbase.master.info.port”: “${HBASE_MASTER.ALLOCATED_PORT}”</p>
+<p><a href="application_definition.md">Application Definition</a> describes how to advertise arbitrary set of properties that are dynamically finalized when application is activated.</p>
+<p><b>configuraing an app for ganglia metrics</b></p>
+<p>There is no set guideline for doing so. How an application emits metrics and how the metrics are emitted to the right place is completely defined by the application. In the following example, we hso how HBase app is configured to emit metrics to a ganglia server.</p>
+<p>Ganglia server lifecycle is not controlled by the app instance. So the app instance only needs to know where to emit the metrics. This is achieved by three global variables</p>
+
+<ul>
+
+<li>“site.global.ganglia_server_host”: “gangliaserver.my.org”</li>
+
+<li>“site.global.ganglia_server_port”: “8663”</li>
+
+<li>“site.global.ganglia_server_id”: “HBaseApplicationCluster3”</li>
+</ul>
+<p>All three variable values are user provided. It is also expected that a gmond server is available on host gangliaserver.my.org listening for metrics at port 8663 and is named “HBaseApplicationCluster3”. Its the reponsibility of the ganglia server admin to ensure that the server is unique and is only receving metrics from the application instance.</p>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container">
+ <div class="row span12">Copyright © 2014
+ <a href="http://www.apache.org/">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+ <?xml version="1.0" encoding="UTF-8"?>
+<div class="row-fluid">Apache Slider, Slider, Apache, and the Apache Incubator logo are trademarks of The Apache Software Foundation.</div>
+
+ <p id="poweredBy" class="pull-right">
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../images/logos/maven-feather.png" />
+ </a>
+ </p>
+
+ </div>
+ </footer>
+ </body>
+</html>
Propchange: incubator/slider/site/content/slider_specs/application_instance_configuration.html
------------------------------------------------------------------------------
svn:eol-style = native
Added: incubator/slider/site/content/slider_specs/application_needs.html
URL: http://svn.apache.org/viewvc/incubator/slider/site/content/slider_specs/application_needs.html?rev=1598665&view=auto
==============================================================================
--- incubator/slider/site/content/slider_specs/application_needs.html (added)
+++ incubator/slider/site/content/slider_specs/application_needs.html Fri May 30 16:47:13 2014
@@ -0,0 +1,316 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2014-05-30
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20140530" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache Slider 0.30 (incubating) - </title>
+ <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../css/site.css" />
+ <link rel="stylesheet" href="../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+ </head>
+ <body class="topBarEnabled">
+
+
+
+
+
+ <div id="topbar" class="navbar navbar-fixed-top ">
+ <div class="navbar-inner">
+ <div class="container"><div class="nav-collapse">
+
+
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Documentation <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li class="dropdown-submenu">
+ <a href="../project-info.html" title="Project Information">Project Information</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../index.html" title="About">About</a>
+</li>
+ <li> <a href="../plugin-management.html" title="Plugin Management">Plugin Management</a>
+</li>
+ <li> <a href="../distribution-management.html" title="Distribution Management">Distribution Management</a>
+</li>
+ <li> <a href="../dependency-info.html" title="Dependency Information">Dependency Information</a>
+</li>
+ <li> <a href="../dependency-convergence.html" title="Dependency Convergence">Dependency Convergence</a>
+</li>
+ <li> <a href="../source-repository.html" title="Source Repository">Source Repository</a>
+</li>
+ <li> <a href="../mail-lists.html" title="Mailing Lists">Mailing Lists</a>
+</li>
+ <li> <a href="../issue-tracking.html" title="Issue Tracking">Issue Tracking</a>
+</li>
+ <li> <a href="../integration.html" title="Continuous Integration">Continuous Integration</a>
+</li>
+ <li> <a href="../plugins.html" title="Project Plugins">Project Plugins</a>
+</li>
+ <li> <a href="../license.html" title="Project License">Project License</a>
+</li>
+ <li> <a href="../modules.html" title="Project Modules">Project Modules</a>
+</li>
+ <li> <a href="../dependency-management.html" title="Dependency Management">Dependency Management</a>
+</li>
+ <li> <a href="../team-list.html" title="Project Team">Project Team</a>
+</li>
+ <li> <a href="../project-summary.html" title="Project Summary">Project Summary</a>
+</li>
+ <li> <a href="../dependencies.html" title="Dependencies">Dependencies</a>
+</li>
+ </ul>
+ </li>
+
+ <li class="dropdown-submenu">
+ <a href="../project-reports.html" title="Project Reports">Project Reports</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../surefire-report.html" title="Surefire Report">Surefire Report</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documents <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../getting_started.html" title="Getting Started">Getting Started</a>
+</li>
+
+ <li> <a href="../manpage.html" title="manpage">manpage</a>
+</li>
+
+ <li> <a href="../troubleshooting.html" title="Troubleshooting">Troubleshooting</a>
+</li>
+
+ <li> <a href="../architecture/index.html" title="Architecture">Architecture</a>
+</li>
+
+ <li> <a href="../developing/index.html" title="Developing">Developing</a>
+</li>
+
+ <li> <a href="../exitcodes.html" title="Exitcodes">Exitcodes</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+ <div class="container">
+ <div id="banner">
+ <div class="pull-left">
+ <a href=".././" id="bannerLeft">
+ <h2>Apache Slider (incubating)</h2>
+ </a>
+ </div>
+ <div class="pull-right"> <div id="bannerRight">
+ <img src="http://incubator.apache.org/images/apache-incubator-logo.png" />
+ </div>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li id="publishDate">Last Published: 2014-05-30</li>
+
+
+
+ <li id="projectVersion" class="pull-right">Version: 0.30</li>
+
+ </ul>
+ </div>
+
+
+
+ <div id="bodyColumn" >
+
+ <!-- -
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. --><h1>Apache Slider’s needs of an application</h1>
+<p>Slider installs and runs applications in a YARN cluster -applications that do not need to be written for YARN. </p>
+<p>What they do need to be is deployable by Slider, which means installable by YARN, configurable by Slider, and, finally, executable by YARN. YARN will kill the executed process when destroying a container, so the deployed application must expect this to happen and be able to start up from a kill-initiated shutdown without any manual recovery process.</p>
+<p>They need to locate each other dynamically, both at startup and during execution, because the location of processes will be unknown at startup, and may change due to server and process failures. </p>
+<div class="section">
+<h2>Must<a name="Must"></a></h2>
+
+<ul>
+
+<li>
+<p>Install and run from a tarball -and be run from a user that is not root.</p></li>
+
+<li>
+<p>Be self contained or have all dependencies pre-installed.</p></li>
+
+<li>
+<p>Support dynamic discovery of nodes -such as via ZK.</p></li>
+
+<li>
+<p>Nodes to rebind themselves dynamically -so if nodes are moved, the application can continue</p></li>
+
+<li>
+<p>Handle kill as a normal shutdown mechanism.</p></li>
+
+<li>
+<p>Support multiple instances of the application running in the same cluster, with processes from different application instances sharing the same servers.</p></li>
+
+<li>
+<p>Operate correctly when more than one role instance in the application is deployed on the same physical host. (If YARN adds anti-affinity options in container requests this will no longer be a requirement)</p></li>
+
+<li>
+<p>Dynamically allocate any RPC or web ports -such as supporting 0 as the number of the port to listen on in configuration options.</p></li>
+
+<li>
+<p>Be trusted. YARN does not run code in a sandbox.</p></li>
+
+<li>
+<p>If it talks to HDFS or other parts of Hadoop, be built against/ship with libaries compatible with the version of Hadoop running on the cluster.</p></li>
+
+<li>
+<p>Store persistent data in HDFS (directly or indirectly) with the exact storage location configurable. Specifically: not to the local filesystem, and not in a hard coded location such as <tt>hdfs://app/data</tt>. Slider creates per-instance directories for persistent data.</p></li>
+
+<li>
+<p>Be configurable as to where any configuration directory is (or simply relative to the tarball). The application must not require it to be in a hard-coded location such as <tt>/etc</tt>.</p></li>
+
+<li>
+<p>Not have a fixed location for log output -such as <tt>/var/log/something</tt></p></li>
+
+<li>
+<p>Run until explicitly terminated. Slider treats an application termination (which triggers a container release) as a failure -and reacts to it by restarting the container.</p></li>
+</ul></div>
+<div class="section">
+<h2>MUST NOT<a name="MUST_NOT"></a></h2>
+
+<ul>
+
+<li>Require human intervention at startup or termination.</li>
+</ul></div>
+<div class="section">
+<h2>SHOULD<a name="SHOULD"></a></h2>
+<p>These are the features that we’d like from a service:</p>
+
+<ul>
+
+<li>
+<p>Publish the actual RPC and HTTP ports in a way that can be picked up, such as via ZK or an admin API.</p></li>
+
+<li>
+<p>Be configurable via the standard Hadoop mechanisms: text files and XML configuration files. If not, custom parsers/configuration generators will be required.</p></li>
+
+<li>
+<p>Support an explicit parameter to define the configuration directory.</p></li>
+
+<li>
+<p>Take late bindings params via -D args or similar</p></li>
+
+<li>
+<p>Be possible to exec without running a complex script, so that process inheritance works everywhere, including (for testing) OS/X</p></li>
+
+<li>
+<p>Provide a way for Slider to get list of nodes in cluster and status. This will let Slider detect failed worker nodes and react to it.</p></li>
+
+<li>
+<p>FUTURE: If a graceful decommissioning is preferred, have an RPC method that a Slider provider can call to invoke this.</p></li>
+
+<li>
+<p>Be location aware from startup. Example: worker nodes to be allocated tables to serve based on which tables are stored locally/in-rack, rather than just randomly. This will accelerate startup time.</p></li>
+
+<li>
+<p>Support simple liveness probes (such as an HTTP GET operations).</p></li>
+
+<li>
+<p>Return a well documented set of exit codes, so that failures can be propagated and understood.</p></li>
+
+<li>
+<p>Support cluster size flexing: the dynamic addition and removal of nodes.</p></li>
+</ul>
+
+<ul>
+
+<li>Support a management platform such as Apache Ambari -so that the operational state of a Slider application can be monitored.</li>
+</ul></div>
+<div class="section">
+<h2>MAY<a name="MAY"></a></h2>
+
+<ul>
+
+<li>
+<p>Include a single process that will run at a fixed location and whose termination can trigger application termination. Such a process will be executed in the same container as the Slider AM, and so known before all other containers are requested. If a live cluster is unable to handle restart/migration of such a process, then the Slider application will be unable to handle Slider AM restarts.</p></li>
+
+<li>
+<p>Ideally: report on load/cost of decommissioning. E.g amount of data; app load. </p></li>
+</ul></div>
+<div class="section">
+<h2>MAY NOT<a name="MAY_NOT"></a></h2>
+
+<ul>
+
+<li>
+<p>Be written for YARN.</p></li>
+
+<li>
+<p>Be (pure) Java. If the tarball contains native binaries for the cluster’s hardware & OS, they should be executable.</p></li>
+
+<li>
+<p>Be dynamically reconfigurable, except for the special requirement of handling movement of manager/peer containers in an application-specific manner.</p></li>
+</ul></div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container">
+ <div class="row span12">Copyright © 2014
+ <a href="http://www.apache.org/">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+ <?xml version="1.0" encoding="UTF-8"?>
+<div class="row-fluid">Apache Slider, Slider, Apache, and the Apache Incubator logo are trademarks of The Apache Software Foundation.</div>
+
+ <p id="poweredBy" class="pull-right">
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../images/logos/maven-feather.png" />
+ </a>
+ </p>
+
+ </div>
+ </footer>
+ </body>
+</html>
Propchange: incubator/slider/site/content/slider_specs/application_needs.html
------------------------------------------------------------------------------
svn:eol-style = native
Added: incubator/slider/site/content/slider_specs/application_package.html
URL: http://svn.apache.org/viewvc/incubator/slider/site/content/slider_specs/application_package.html?rev=1598665&view=auto
==============================================================================
--- incubator/slider/site/content/slider_specs/application_package.html (added)
+++ incubator/slider/site/content/slider_specs/application_package.html Fri May 30 16:47:13 2014
@@ -0,0 +1,332 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2014-05-30
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20140530" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache Slider 0.30 (incubating) - </title>
+ <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../css/site.css" />
+ <link rel="stylesheet" href="../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+ </head>
+ <body class="topBarEnabled">
+
+
+
+
+
+ <div id="topbar" class="navbar navbar-fixed-top ">
+ <div class="navbar-inner">
+ <div class="container"><div class="nav-collapse">
+
+
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Documentation <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li class="dropdown-submenu">
+ <a href="../project-info.html" title="Project Information">Project Information</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../index.html" title="About">About</a>
+</li>
+ <li> <a href="../plugin-management.html" title="Plugin Management">Plugin Management</a>
+</li>
+ <li> <a href="../distribution-management.html" title="Distribution Management">Distribution Management</a>
+</li>
+ <li> <a href="../dependency-info.html" title="Dependency Information">Dependency Information</a>
+</li>
+ <li> <a href="../dependency-convergence.html" title="Dependency Convergence">Dependency Convergence</a>
+</li>
+ <li> <a href="../source-repository.html" title="Source Repository">Source Repository</a>
+</li>
+ <li> <a href="../mail-lists.html" title="Mailing Lists">Mailing Lists</a>
+</li>
+ <li> <a href="../issue-tracking.html" title="Issue Tracking">Issue Tracking</a>
+</li>
+ <li> <a href="../integration.html" title="Continuous Integration">Continuous Integration</a>
+</li>
+ <li> <a href="../plugins.html" title="Project Plugins">Project Plugins</a>
+</li>
+ <li> <a href="../license.html" title="Project License">Project License</a>
+</li>
+ <li> <a href="../modules.html" title="Project Modules">Project Modules</a>
+</li>
+ <li> <a href="../dependency-management.html" title="Dependency Management">Dependency Management</a>
+</li>
+ <li> <a href="../team-list.html" title="Project Team">Project Team</a>
+</li>
+ <li> <a href="../project-summary.html" title="Project Summary">Project Summary</a>
+</li>
+ <li> <a href="../dependencies.html" title="Dependencies">Dependencies</a>
+</li>
+ </ul>
+ </li>
+
+ <li class="dropdown-submenu">
+ <a href="../project-reports.html" title="Project Reports">Project Reports</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../surefire-report.html" title="Surefire Report">Surefire Report</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documents <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../getting_started.html" title="Getting Started">Getting Started</a>
+</li>
+
+ <li> <a href="../manpage.html" title="manpage">manpage</a>
+</li>
+
+ <li> <a href="../troubleshooting.html" title="Troubleshooting">Troubleshooting</a>
+</li>
+
+ <li> <a href="../architecture/index.html" title="Architecture">Architecture</a>
+</li>
+
+ <li> <a href="../developing/index.html" title="Developing">Developing</a>
+</li>
+
+ <li> <a href="../exitcodes.html" title="Exitcodes">Exitcodes</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+ <div class="container">
+ <div id="banner">
+ <div class="pull-left">
+ <a href=".././" id="bannerLeft">
+ <h2>Apache Slider (incubating)</h2>
+ </a>
+ </div>
+ <div class="pull-right"> <div id="bannerRight">
+ <img src="http://incubator.apache.org/images/apache-incubator-logo.png" />
+ </div>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li id="publishDate">Last Published: 2014-05-30</li>
+
+
+
+ <li id="projectVersion" class="pull-right">Version: 0.30</li>
+
+ </ul>
+ </div>
+
+
+
+ <div id="bodyColumn" >
+
+ <!-- -
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. --><h1>Apache Slider App Package</h1>
+<p>All application artifacts, app definition, app configuration, scripts are packaged into a structured single package that can be handed off to any YARN application deployment tool including Slider</p>
+<div class="section">
+<h2>Overall Structure<a name="Overall_Structure"></a></h2>
+<p>App package is a zip package containing all application artifacts. App package contains the following items:</p>
+
+<ul>
+
+<li>
+<p><b>app definition file</b> application structure, content, definition, supported platforms, version, etc.</p></li>
+
+<li>
+<p><b>default configurations folder</b> various configurations and configuration files associated with the application</p></li>
+
+<li>
+<p><b>cmd_impl folder</b> management operations for the application/component</p></li>
+
+<li>
+<p><b>scripts folder</b> various scripts that implement management operations</p></li>
+
+<li>
+<p><b>templates folder</b> various templates used by the application</p></li>
+
+<li>
+<p><b>files folder</b> other scripts, txt files, tarballs, etc.</p></li>
+</ul>
+<p><img src="../../resources/images/app_package_sample_04.png" alt="Image" /></p>
+<p>The example above shows a semi-expanded view of an application “HBASE-YARN-APP” and the package structure for OOZIE command scripts.</p></div>
+<div class="section">
+<h2>app definition<a name="app_definition"></a></h2>
+<p>App definition is a file named “metainfo.xml”. The file contains application definition as described in <a href="application_definition.md">Application Definition</a>. </p></div>
+<div class="section">
+<h2>default configurations<a name="default_configurations"></a></h2>
+<p>This folder consists of various config files containing default configuration as described in <a href="application_configuration.md">App Configuration</a>.</p></div>
+<div class="section">
+<h2>package folder<a name="package_folder"></a></h2>
+<p>package includes the “implementation” of all management operations. The folders within are divided into scripts, templates, and files.</p>
+<div class="section">
+<h3>scripts folder<a name="scripts_folder"></a></h3>
+<p>Scripts are the implementation of management operations. There are five default operations and a composite operation. “restart” can be redefined to have a custom implementation.</p>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>install</p></li>
+
+<li>
+<p>configure</p></li>
+
+<li>
+<p>start</p></li>
+
+<li>
+<p>stop</p></li>
+
+<li>
+<p>status</p></li>
+
+<li>
+<p>restart (by default calls stop + start)</p></li>
+</ol>
+<p>The script specified in the metainfo is expected to understand the command. It can choose to call other scripts based on how the application author organizes the code base. For example:</p>
+
+<div class="source">
+<pre>class OozieServer(Script):
+ def install(self, env):
+ self.install_packages(env)
+
+ def configure(self, env):
+ import params
+ env.set_params(params)
+ oozie(is_server=True)
+
+ def start(self, env):
+ import params
+ env.set_params(params)
+ self.configure(env)
+ oozie_service(action='start')
+
+ def stop(self, env):
+ import params
+ env.set_params(params)
+ oozie_service(action='stop')
+
+ def status(self, env):
+ import status_params
+ env.set_params(status_params)
+ check_process_status(status_params.pid_file)
+</pre></div>
+<p>The scripts are invoked in the following manner:</p>
+
+<div class="source">
+<pre>python SCRIPT COMMAND JSON_FILE PACKAGE_ROOT STRUCTURED_OUT_FILE
+</pre></div>
+
+<ul>
+
+<li>
+<p>SCRIPT is the top level script that implements the commands for the component.</p></li>
+
+<li>
+<p>COMMAND is one of the six commands listed above or can be a custom command as defined in Application Definition</p></li>
+
+<li>
+<p>JSON_FILE includes all configuration parameters and the values</p></li>
+
+<li>
+<p>PACKAGE_ROOT is the root folder of the package. From this folder, its possible to access files, scripts, templates, packages (e.g. tarballs), etc. The App author has complete control over the structure of the package</p></li>
+
+<li>
+<p>STRUCTURED_OUT_FILE is the file where the script can output structured data. The management infrastructure is expected to automatically reports back STD_OUT and STD_ERR.</p></li>
+</ul>
+<p>A separate document (link TBD) discusses how the scripts are developed and the structure of the JSON_FILE containing the parameters.</p></div>
+<div class="section">
+<h3>templates folder<a name="templates_folder"></a></h3>
+<p>templates are configurable text files that are NOT regular config files. <i>A library has been developed that can materialize a complete site configuration file from a property bag and therefore are not generated from templates.</i> Other files such as env sh files, log4j properties file, etc. may be derived from a template. Again, the implementor can choose to create these files from scratch and not use templates. The following operations are allowed during template expansion:</p>
+
+<ul>
+
+<li>
+<p>variable expansion</p></li>
+
+<li>
+<p>if condition</p></li>
+
+<li>
+<p>for loop</p></li>
+
+<li>
+<p>…</p></li>
+</ul>
+<p>Sample template file for dfs.exclude file to list excluded/decommissioned hosts. hdfs_exclude_files in the property defined in params.py which is populated from config parameters defined in JSON_FILE.</p>
+
+<div class="source">
+<pre>{% if hdfs_exclude_file %}
+{% for host in hdfs_exclude_file %}
+{{host}}
+{% endfor %}
+{% endif %}
+</pre></div></div>
+<div class="section">
+<h3>files folder<a name="files_folder"></a></h3>
+<p>files is a directory to store any other files that are needed for management operations. Sample files stored here are tarballs used to install the application, shell scripts used by various operations.</p></div></div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container">
+ <div class="row span12">Copyright © 2014
+ <a href="http://www.apache.org/">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+ <?xml version="1.0" encoding="UTF-8"?>
+<div class="row-fluid">Apache Slider, Slider, Apache, and the Apache Incubator logo are trademarks of The Apache Software Foundation.</div>
+
+ <p id="poweredBy" class="pull-right">
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../images/logos/maven-feather.png" />
+ </a>
+ </p>
+
+ </div>
+ </footer>
+ </body>
+</html>
Propchange: incubator/slider/site/content/slider_specs/application_package.html
------------------------------------------------------------------------------
svn:eol-style = native
Added: incubator/slider/site/content/slider_specs/canonical_scenarios.html
URL: http://svn.apache.org/viewvc/incubator/slider/site/content/slider_specs/canonical_scenarios.html?rev=1598665&view=auto
==============================================================================
--- incubator/slider/site/content/slider_specs/canonical_scenarios.html (added)
+++ incubator/slider/site/content/slider_specs/canonical_scenarios.html Fri May 30 16:47:13 2014
@@ -0,0 +1,394 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2014-05-30
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20140530" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache Slider 0.30 (incubating) - </title>
+ <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../css/site.css" />
+ <link rel="stylesheet" href="../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+ </head>
+ <body class="topBarEnabled">
+
+
+
+
+
+ <div id="topbar" class="navbar navbar-fixed-top ">
+ <div class="navbar-inner">
+ <div class="container"><div class="nav-collapse">
+
+
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Documentation <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li class="dropdown-submenu">
+ <a href="../project-info.html" title="Project Information">Project Information</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../index.html" title="About">About</a>
+</li>
+ <li> <a href="../plugin-management.html" title="Plugin Management">Plugin Management</a>
+</li>
+ <li> <a href="../distribution-management.html" title="Distribution Management">Distribution Management</a>
+</li>
+ <li> <a href="../dependency-info.html" title="Dependency Information">Dependency Information</a>
+</li>
+ <li> <a href="../dependency-convergence.html" title="Dependency Convergence">Dependency Convergence</a>
+</li>
+ <li> <a href="../source-repository.html" title="Source Repository">Source Repository</a>
+</li>
+ <li> <a href="../mail-lists.html" title="Mailing Lists">Mailing Lists</a>
+</li>
+ <li> <a href="../issue-tracking.html" title="Issue Tracking">Issue Tracking</a>
+</li>
+ <li> <a href="../integration.html" title="Continuous Integration">Continuous Integration</a>
+</li>
+ <li> <a href="../plugins.html" title="Project Plugins">Project Plugins</a>
+</li>
+ <li> <a href="../license.html" title="Project License">Project License</a>
+</li>
+ <li> <a href="../modules.html" title="Project Modules">Project Modules</a>
+</li>
+ <li> <a href="../dependency-management.html" title="Dependency Management">Dependency Management</a>
+</li>
+ <li> <a href="../team-list.html" title="Project Team">Project Team</a>
+</li>
+ <li> <a href="../project-summary.html" title="Project Summary">Project Summary</a>
+</li>
+ <li> <a href="../dependencies.html" title="Dependencies">Dependencies</a>
+</li>
+ </ul>
+ </li>
+
+ <li class="dropdown-submenu">
+ <a href="../project-reports.html" title="Project Reports">Project Reports</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../surefire-report.html" title="Surefire Report">Surefire Report</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documents <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../getting_started.html" title="Getting Started">Getting Started</a>
+</li>
+
+ <li> <a href="../manpage.html" title="manpage">manpage</a>
+</li>
+
+ <li> <a href="../troubleshooting.html" title="Troubleshooting">Troubleshooting</a>
+</li>
+
+ <li> <a href="../architecture/index.html" title="Architecture">Architecture</a>
+</li>
+
+ <li> <a href="../developing/index.html" title="Developing">Developing</a>
+</li>
+
+ <li> <a href="../exitcodes.html" title="Exitcodes">Exitcodes</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+ <div class="container">
+ <div id="banner">
+ <div class="pull-left">
+ <a href=".././" id="bannerLeft">
+ <h2>Apache Slider (incubating)</h2>
+ </a>
+ </div>
+ <div class="pull-right"> <div id="bannerRight">
+ <img src="http://incubator.apache.org/images/apache-incubator-logo.png" />
+ </div>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li id="publishDate">Last Published: 2014-05-30</li>
+
+
+
+ <li id="projectVersion" class="pull-right">Version: 0.30</li>
+
+ </ul>
+ </div>
+
+
+
+ <div id="bodyColumn" >
+
+ <!-- -
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. --><h1>Apache Slider Guidelines for Clients and Client Applications</h1>
+<p>This document will define the canonical scenarios for the deployment and management of Slider hosted applications. It will define the types of applications supported, the sequence of events for deploying the application types, and the management facilities exposed for the deployed applications.</p>
+<div class="section">
+<h2>Deployable Application Types<a name="Deployable_Application_Types"></a></h2>
+<p>The server-side components of an application (alternatively referred to as the application components) will be deployed and managed using a fairly uniform mechanism. However, applications can be distinguished by their associated client implementation. Specifically, each different client application type can yield different development, deployment, and management approaches. There are two primary application client types:</p>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p><b>New application client</b> - the application deployed to Slider will interact with a newly coded application client leveraging the Slider supplied client API and management facilities. The client API will make use of distributed management facilities such as Zookeeper to provide the application client with the configuration information required for invoking remote application components. For example, the application client (or an application hosted component, e.g. a web application) will leverage the API to lookup the appropriate host(s) and port(s) for RPC based communication. Alternatively, if annotation libraries or an “app container” environment is provided, the appropriate values will be injected into the client process.</p></li>
+
+<li>
+<p><b>Existing, configurable client</b> - the application client predates deployment in the Slider environment, but is implemented in a manner that can be integrated with the Slider application client support facilities (APIs etc). This case is probably very similar in nature to the new application client in terms of the mechanisms it will use for component discovery, but is distinguished by the fact that it’s development pre-dates Slider. There are two possible variants of application client in this category:</p></li>
+
+<li>
+<p>A client that is static - the client is dependent on existing configuration properties to communicate with master components and the existing code can not been altered (at least in the short term). This type of client would require a support infrastructure that may be relatively complex (e.g. templating of configuration files, proxying of server components).</p></li>
+
+<li>
+<p>A client that can be enhanced - a client that can have its code altered can leverage a number of mechanisms (REST APIs, Zookeeper, a provided client discover API) to obtain the information required to invoke the master components of the application.</p></li>
+</ol></div>
+<div class="section">
+<h2>Deployment Scenarios<a name="Deployment_Scenarios"></a></h2>
+<p>There are two primary deployment mechanisms to examine: application component and client-side (application client) component deployment.</p></div>
+<div class="section">
+<h2>Application Component Deployment<a name="Application_Component_Deployment"></a></h2>
+<p>Applications generally are composed of one or more components. In the deployment steps below, be advised that there may be a need to repeat some of the configuration/definition steps for each component.</p>
+<p>The process of deploying applications (specifically, the non-client components of an application) is:</p>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Compose an application package that contains:</p></li>
+
+<li>
+<p>An application definition that provides the following items:</p>
+
+<ol style="list-style-type: decimal">
+
+<li>name and version of application</li>
+ </ol>
+
+<ol style="list-style-type: decimal">
+
+<li>component type(s) and role(s)</li>
+ </ol>
+
+<ol style="list-style-type: decimal">
+
+<li>system requirements (RAM, CPUs, disk space etc)</li>
+ </ol>
+
+<ol style="list-style-type: decimal">
+
+<li>ports required for RPC, UI</li>
+ </ol>
+
+<ol style="list-style-type: decimal">
+
+<li>software dependencies (HDP deployed services required in the cluster, JDK versions, python versions, etc)</li>
+ </ol>
+
+<ol style="list-style-type: decimal">
+
+<li>configurable properties including:</li>
+ </ol>
+
+<div class="source">
+<pre> 1. application-specific properties. If the properties designate port numbers, the general recommendation would be to set them to zero to allow for the system to assign a dynamically selected port number that subsequently can be published to zookeeper. Other properties designating remote components running on other hosts may need some additional support (floating IPs, discovery APIs, etc). The processing of these properties by the infrastructure requires agreed upon mechanisms for identification (e.g. all properties beginning with the application name are passed through as application properties with the application name prefix stripped)
+
+ 2. Slider-specific properties. The following Slider properties are currently available:
+
+ 1. yarn.memory
+
+ 2. Environment variables specified with a "env." prefix (e.g. env.MALLOC_ARENA_MAX)
+
+ 3. role.instances
+
+ 4. role.name
+
+ 5. yarn.vcores
+</pre></div></li>
+
+<li>
+<p>Other application artifacts (binaries etc)</p></li>
+
+<li>
+<p>Install the application</p></li>
+
+<li>
+<p>The application package will be made available in a designated HDFS location</p></li>
+
+<li>
+<p>If there is a managed application client component it will be deployed to selected nodes as defined in the cluster specification.</p></li>
+
+<li>
+<p>Slider interacts with yarn (resource manager and node manager(s)) to populate the local resources on the designated nodes.</p></li>
+
+<li>
+<p>Some properties important for facilitating remote interaction with the deployed components are advertised via zookeeper (though this may actually take place during component start up as the assignment of ports is a late-binding operation. Alternatively, developers may be encouraged to store these values in a registry rather than as direct-application configuration properties).</p></li>
+</ol></div>
+<div class="section">
+<h2>Client Application Deployment<a name="Client_Application_Deployment"></a></h2>
+<p>Although application clients are components that are deployed using mechanisms similar to other application components (especially in the managed case), there are a number of features that distinguish them:</p>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p><b>configuration</b> - client applications generally require some information (host names, ports, etc) to ascertain the correct networking values required to communicate with the application’s server components. In order to work in a yarn deployment, this configuration may need to be manipulated to allow proper operation (e.g. the configuration files updated with correct values, the configuration properties ascertained dynamically from services such as Zookeeper)</p></li>
+
+<li>
+<p><b>execution context </b>- it may be necessary to provide an execution environment for application clients that allows for discovery mechanisms (dependency injection, annotation libraries, etc)</p></li>
+</ol>
+<p>For each of these application client types there are two possible deployment modes:</p>
+
+<ul>
+
+<li><b>Managed</b> - the application client is deployed via Slider mechanisms. Clients, in this context, differ from the other application components in that they are not running, daemon processes. However, in a managed environment there is the expectation that the appropriate binaries and application elements will be distributed to the designated client hosts, and the configuration on those hosts will be updated to allow for execution of requests to the application’s master/server components. Therefore, client components should be defined in the application specification as elements that the management infrastructure supports (Figure 1).</li>
+</ul>
+<p><img src="../images/managed_client.png" alt="Image" /> Figure 1 - Managed Application Client and associated Slider Application</p>
+
+<ul>
+
+<li><b>Unmanaged</b> - the application client is run as a process outside of Slider/yarn, although it may leverage Slider provided libraries that allow for server component discovery etc (Figure 2). These libraries would primarily be client bindings providing access to the registry leveraged by Slider (e.g. Java and python bindings to Zookeeper)</li>
+</ul>
+<p><img src="../images/unmanaged_client.png" alt="Image" /> Figure 2 - Unmanaged Application Client and associated Slider Application</p>
+<div class="section">
+<h3>Managed Application Client<a name="Managed_Application_Client"></a></h3>
+<p>A managed application client is a component defined as part of the Slider/yarn application (i.e. it is part of the application definition presented to Slider). As such, it is deployed and managed via standard Slider/yarn mechanisms. This type of application client is more than likely already configured and written to work in a yarn environment.</p>
+<p>There are two primary needs to be met for a properly functioning client:</p>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p><b>Discovery</b> - as a client, it is important that the client application retrieve the information it requires in order to communicate with the remote application components. As application components are spawned they (or the associated container agent) will advertise the relevant information using zookeeper. It will be up to the client (or the associated Slider client library) to contact zookeeper and retrieve the requested information.</p></li>
+
+<li>
+<p><b>Configuration</b> - there may be use cases in which a large number of configuration items are required by the client for its processing. In such cases it is more appropriate for a client to perform a bulk download of the application component(s) configuration as a JSON or XML document (via zookeeper or Slider-app comm?)</p></li>
+</ol>
+<p>Whether ascertained via discovery or bulk configuration retrieval, the attributes that the client obtains will more than likely need to be populated into the client’s configuration files. Therefore, a templating facility or the like should be provided to allow for such configuration file manipulation.</p></div>
+<div class="section">
+<h3>Unmanaged Application Client<a name="Unmanaged_Application_Client"></a></h3>
+<p>An unmanaged application client is a standalone application that leverages application components deployed into the Slider/yarn environment. It is not possible to predict the deployment mechanisms or network topology employed for an unmanaged application. However, it is feasible to provide some guidance and/or software (APIs etc) to allow for application component discovery and communication.</p></div></div>
+<div class="section">
+<h2>Application Management<a name="Application_Management"></a></h2>
+<p>Post deployment, the Slider infrastructure will provide the requisite set of administrative facilities required for application management, including the ability to start/stop the application, monitor the application, and reconfigure the application. </p>
+<div class="section">
+<h3>General Management<a name="General_Management"></a></h3>
+<p>There is one general management command:</p>
+
+<ul>
+
+<li>
+<p>List Yarn Apps - returns a listing of deployed yarn apps and associated information:</p></li>
+
+<li>
+<p>name and version</p></li>
+
+<li>
+<p>dependencies (required HDP services and versions, etc)</p></li>
+
+<li>
+<p>configuration properties</p></li>
+
+<li>
+<p>components/roles and associated configuration</p></li>
+</ul></div>
+<div class="section">
+<h3>Application/Component Management<a name="ApplicationComponent_Management"></a></h3>
+<p>The following administrative functions are supported for applications:</p>
+
+<ul>
+
+<li>
+<p>Install the application - the installation command will take care of the population of the application resources into the pre-determined application resource directory in HDFS. The required configuration and binary directories will also be created.</p></li>
+
+<li>
+<p>start/thaw the application - Slider/Yarn runtime negotiates and instantiates the number of component containers designated by the cluster description and the components are started.</p></li>
+
+<li>
+<p>stop/freeze the application - similar to stopping, applicaiton (or a subset of their components) can be stopped.</p></li>
+
+<li>
+<p>get application status - the retrieval of application status may take a number of forms, including:</p></li>
+
+<li>
+<p>liveness of service components</p></li>
+
+<li>
+<p>operational metrics (application-level or component-level)</p></li>
+
+<li>
+<p>viewing of logs</p></li>
+
+<li>
+<p>get application configuration - the configuration of application components is retrieved (JSON or XML form)</p></li>
+
+<li>
+<p>get cluster configuration - the cluster configuration is retrieved (number of various application components, associated hosts etc)</p></li>
+
+<li>
+<p>get cluster history</p></li>
+
+<li>
+<p>re-configure cluster</p></li>
+</ul></div></div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container">
+ <div class="row span12">Copyright © 2014
+ <a href="http://www.apache.org/">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+ <?xml version="1.0" encoding="UTF-8"?>
+<div class="row-fluid">Apache Slider, Slider, Apache, and the Apache Incubator logo are trademarks of The Apache Software Foundation.</div>
+
+ <p id="poweredBy" class="pull-right">
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../images/logos/maven-feather.png" />
+ </a>
+ </p>
+
+ </div>
+ </footer>
+ </body>
+</html>
Propchange: incubator/slider/site/content/slider_specs/canonical_scenarios.html
------------------------------------------------------------------------------
svn:eol-style = native