You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@slider.apache.org by bi...@apache.org on 2014/05/30 18:47:16 UTC
svn commit: r1598665 [3/16] - in /incubator/slider/site/content: ./
architecture/ configuration/ css/ developing/ images/ images/logos/
images/profiles/ img/ js/ registry/ release_notes/ slider_specs/
specification/
Added: incubator/slider/site/content/configuration/core.html
URL: http://svn.apache.org/viewvc/incubator/slider/site/content/configuration/core.html?rev=1598665&view=auto
==============================================================================
--- incubator/slider/site/content/configuration/core.html (added)
+++ incubator/slider/site/content/configuration/core.html Fri May 30 16:47:13 2014
@@ -0,0 +1,528 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2014-05-30
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20140530" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache Slider 0.30 (incubating) - </title>
+ <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../css/site.css" />
+ <link rel="stylesheet" href="../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+ </head>
+ <body class="topBarEnabled">
+
+
+
+
+
+ <div id="topbar" class="navbar navbar-fixed-top ">
+ <div class="navbar-inner">
+ <div class="container"><div class="nav-collapse">
+
+
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Documentation <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li class="dropdown-submenu">
+ <a href="../project-info.html" title="Project Information">Project Information</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../index.html" title="About">About</a>
+</li>
+ <li> <a href="../plugin-management.html" title="Plugin Management">Plugin Management</a>
+</li>
+ <li> <a href="../distribution-management.html" title="Distribution Management">Distribution Management</a>
+</li>
+ <li> <a href="../dependency-info.html" title="Dependency Information">Dependency Information</a>
+</li>
+ <li> <a href="../dependency-convergence.html" title="Dependency Convergence">Dependency Convergence</a>
+</li>
+ <li> <a href="../source-repository.html" title="Source Repository">Source Repository</a>
+</li>
+ <li> <a href="../mail-lists.html" title="Mailing Lists">Mailing Lists</a>
+</li>
+ <li> <a href="../issue-tracking.html" title="Issue Tracking">Issue Tracking</a>
+</li>
+ <li> <a href="../integration.html" title="Continuous Integration">Continuous Integration</a>
+</li>
+ <li> <a href="../plugins.html" title="Project Plugins">Project Plugins</a>
+</li>
+ <li> <a href="../license.html" title="Project License">Project License</a>
+</li>
+ <li> <a href="../modules.html" title="Project Modules">Project Modules</a>
+</li>
+ <li> <a href="../dependency-management.html" title="Dependency Management">Dependency Management</a>
+</li>
+ <li> <a href="../team-list.html" title="Project Team">Project Team</a>
+</li>
+ <li> <a href="../project-summary.html" title="Project Summary">Project Summary</a>
+</li>
+ <li> <a href="../dependencies.html" title="Dependencies">Dependencies</a>
+</li>
+ </ul>
+ </li>
+
+ <li class="dropdown-submenu">
+ <a href="../project-reports.html" title="Project Reports">Project Reports</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../surefire-report.html" title="Surefire Report">Surefire Report</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documents <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../getting_started.html" title="Getting Started">Getting Started</a>
+</li>
+
+ <li> <a href="../manpage.html" title="manpage">manpage</a>
+</li>
+
+ <li> <a href="../troubleshooting.html" title="Troubleshooting">Troubleshooting</a>
+</li>
+
+ <li> <a href="../architecture/index.html" title="Architecture">Architecture</a>
+</li>
+
+ <li> <a href="../developing/index.html" title="Developing">Developing</a>
+</li>
+
+ <li> <a href="../exitcodes.html" title="Exitcodes">Exitcodes</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+ <div class="container">
+ <div id="banner">
+ <div class="pull-left">
+ <a href=".././" id="bannerLeft">
+ <h2>Apache Slider (incubating)</h2>
+ </a>
+ </div>
+ <div class="pull-right"> <div id="bannerRight">
+ <img src="http://incubator.apache.org/images/apache-incubator-logo.png" />
+ </div>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li id="publishDate">Last Published: 2014-05-30</li>
+
+
+
+ <li id="projectVersion" class="pull-right">Version: 0.30</li>
+
+ </ul>
+ </div>
+
+
+
+ <div id="bodyColumn" >
+
+ <!-- -
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. --><h1>Apache Slider Core Configuration Specification</h1>
+<div class="section">
+<h2>Terminology<a name="Terminology"></a></h2>
+<p><i>Application</i> A single application, such as an HBase cluster. An application is distribed across the YARN cluster.</p>
+<p><i>Component</i> A single executable part of the larger application. An application may have multiple components, and multiple instances of each component. </p>
+<p><i>YARN</i> Yet Another Resource Negotiator</p>
+<p><i>YARN Resource Requirements</i> The requirements for a YARN resource request. Currently this consists of RAM and CPU requirements.</p>
+<p><i>YARN Container</i>. An allocation portion of a servers resources granted to satisfy the requested YARN resource requirements. A process can be deployed to a container.</p>
+<p><i><tt>resources.json</tt></i>: A file that describes the size of the application in terms of its component requirements: how many, and what their resource requirements are. </p>
+<p><i><tt>application.json</tt></i>: A file that describes the size of the application in terms of its component requirements: how many, and what their resource requirements are. </p></div>
+<div class="section">
+<h2>Structure<a name="Structure"></a></h2>
+<p>Configurations are stored in well-formed JSON files. 1. Text MUST be saved in the UTF-8 format. 1. Duplicate entries MUST NOT occur in any section. 1. The ordering of elements is NOT significant.</p>
+<p>The JSON specification files all have a similar structure</p>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>A <tt>schema</tt> string indicating version. Currently this is temporarily set to</p>
+
+<div class="source">
+<pre>"http://example.org/specification/v2.0.0"
+</pre></div></li>
+
+<li>A global section, <tt>/global</tt> containing string properties</li>
+
+<li>A component section, <tt>/components</tt>.</li>
+
+<li>0 or more sections under <tt>/components</tt> for each component, identified by component name, containing string properties.</li>
+
+<li>0 or 1 section <tt>/metadata</tt> containing arbitrary metadata (such as a description, author, or any other information that is not parsed or processed directly).</li>
+</ol>
+<p>The simplest valid specification file is </p>
+
+<div class="source">
+<pre>{
+ "schema": "http://example.org/specification/v2.0.0",
+
+ "global": {
+ },
+ "components": {
+ }
+}
+</pre></div></div>
+<div class="section">
+<h2>Property inheritance model and <i>resolution</i><a name="Property_inheritance_model_and_resolution"></a></h2>
+<p>There is a simple global to component inheritance model.</p>
+
+<ol style="list-style-type: decimal">
+
+<li>Properties defined in <tt>/global</tt> define parameters across the entire application.</li>
+
+<li>Properties defined a section under <tt>/components</tt> define parameters for a specific component in the application.</li>
+
+<li>All global properties are propagated to each component.</li>
+
+<li>A component section may override any global property.</li>
+
+<li>The final set of configuration properties for a component is the global properties extended and overridden by the global set.</li>
+
+<li>The process of expanding the properties is termed <i>resolution</i>; the <i>resolved</i> specification is the outcome.</li>
+
+<li>There is NO form of explicitly cross-referencing another attribute. This MAY be added in future.</li>
+
+<li>There is NO sharing of information from the different <tt>.json</tt> files in a an application configuration.</li>
+</ol>
+<div class="section">
+<h3>Example<a name="Example"></a></h3>
+<p>Here is an example configuration</p>
+
+<div class="source">
+<pre>{
+ "schema": "http://example.org/specification/v2.0.0",
+
+ "global": {
+ "g1": "a",
+ "g2": "b"
+ },
+ "components": {
+ "simple": {
+ },
+ "master": {
+ "name": "m",
+ "g1": "overridden"
+
+ },
+ "worker": {
+ "name": "w",
+ "g1": "overridden-by-worker",
+ "timeout": "1000"
+ }
+ }
+}
+</pre></div>
+<p>The <tt>/global</tt> section defines two properties</p>
+
+<div class="source">
+<pre>g1="a"
+g2="b"
+</pre></div>
+<p>These are the values visible to any part of the application which is not itself one of the components. </p>
+<p>There are three components defined, <tt>simple</tt>, <tt>master</tt> and <tt>worker</tt>.</p>
+<div class="section">
+<h4>component <tt>simple</tt>:<a name="component_simple:"></a></h4>
+
+<div class="source">
+<pre>g1="a"
+g2="b"
+</pre></div>
+<p>No settings have been defined specifically for the component; the global settings are applied.</p></div>
+<div class="section">
+<h4>component <tt>master</tt>:<a name="component_master:"></a></h4>
+
+<div class="source">
+<pre>name="m",
+g1="overridden"
+g2="b"
+</pre></div>
+<p>A new attribute, <tt>name</tt>, has been defined with the value <tt>"m"</tt>, and the global property <tt>g1</tt> has been overridden with the new value, <tt>"overridden"</tt>. The global property <tt>g2</tt> is passed down unchanged.</p></div>
+<div class="section">
+<h4>component <tt>worker</tt>:<a name="component_worker:"></a></h4>
+
+<div class="source">
+<pre>name="w",
+g1="overridden-by-worker"
+g2="b"
+timeout: "1000"
+</pre></div>
+<p>A new attribute, <tt>name</tt>, has been defined with the value <tt>"w"</tt>, and another, <tt>timeout</tt>, value “1000”. </p>
+<p>The global property <tt>g1</tt> has been overridden with the new value, <tt>"overridden-by-worker"</tt>.</p>
+<p>The global property <tt>g2</tt> is passed down unchanged.</p>
+<p>This example shows some key points about the design</p>
+
+<ul>
+
+<li>each component gets its own map of properties, which is independent from that of other components.</li>
+
+<li>all global properties are either present or overridden by a new value. They can not be “undefined”</li>
+
+<li>new properties defined in a component are not visible to any other component.</li>
+</ul>
+<p>The final <i>resolved</i> model is as follows</p>
+
+<div class="source">
+<pre>{
+ "schema": "http://example.org/specification/v2.0.0",
+
+ "global": {
+ "g1": "a",
+ "g2": "b"
+ },
+ "components": {
+ "simple": {
+ "g1": "a",
+ "g2": "b"
+ },
+ "master": {
+ "name": "m",
+ "g1": "overridden",
+ "g2": "b"
+ },
+ "worker": {
+ "name": "m",
+ "g1": "overridden-by-worker",
+ "g2": "b",
+ "timeout": "1000"
+ }
+ }
+}
+</pre></div>
+<p>This the specification JSON that would have generate exactly the same result as in the example, without any propagation of data from the global section to individual components. </p>
+<p>Note that a resolved specification can still have the resolution operation applied to it -it just does not have any effect.</p></div></div></div>
+<div class="section">
+<h2>Metadata<a name="Metadata"></a></h2>
+<p>The metadata section can contain arbitrary string values for use in diagnostics and by other applications.</p>
+<p>To avoid conflict with other applications, please use a unique name in strings, such as java-style package names.</p>
+<h1>Resource Requirements: <tt>resources.json</tt></h1>
+<p>This file declares the resource requirements for YARN for the components of an application.</p>
+<p><tt>instances</tt>: the number of instances of a role desired. <tt>yarn.vcores</tt>: number of “virtual” required by a component. <tt>yarn.memory</tt>: the number of megabytes required by a component.</p>
+
+<div class="source">
+<pre>{
+ "schema": "http://example.org/specification/v2.0.0",
+
+ "metadata": {
+ "description": "example of a resources file"
+ },
+
+ "global": {
+ "yarn.vcores": "1",
+ "yarn.memory": "512"
+ },
+
+ "components": {
+ "master": {
+ "instances": "1",
+ "yarn.memory": "1024"
+ },
+ "worker": {
+ "instances":"5"
+ }
+ }
+}
+</pre></div>
+<p>The resolved file would be</p>
+
+<div class="source">
+<pre>{
+ "schema": "http://example.org/specification/v2.0.0",
+
+ "metadata": {
+ "description": "example of a resources file"
+ },
+
+ "global": {
+ "yarn.vcores": "1",
+ "yarn.memory": "512"
+ },
+
+ "components": {
+ "master": {
+ "instances": "1",
+ "yarn.vcores": "1",
+ "yarn.memory": "1024"
+ },
+ "worker": {
+ "instances":"5",
+ "yarn.vcores": "1",
+ "yarn.memory": "512"
+ }
+ }
+}
+</pre></div>
+<p>This declares this deployment of the application to consist of one instance of the master component, using 1 vcore and 1024MB of RAM, and five worker components each using one vcore and 512 MB of RAM.</p></div>
+<div class="section">
+<h2>Internal information, <tt>internal.json</tt><a name="Internal_information_internal.json"></a></h2>
+<p>This contains internal data related to the deployment -it is not intended for manual editing.</p>
+<p>There MAY be a component, <tt>diagnostics</tt>. If defined, its content contains diagnostic information for support calls, and MUST NOT be interpreted during application deployment, (though it may be included in the generation of diagnostics reports)</p>
+
+<div class="source">
+<pre>{
+ "schema": "http://example.org/specification/v2.0.0",
+
+ "metadata": {
+ "description": "Internal configuration DO NOT EDIT"
+ },
+ "global": {
+ "name": "small_cluster",
+ "application": "hdfs://cluster:8020/apps/hbase/v/1.0.0/application.tar"
+ },
+ "components": {
+
+ "diagnostics": {
+ "create.hadoop.deployed.info": "(release-2.3.0) @dfe463",
+ "create.hadoop.build.info": "2.3.0",
+ "create.time.millis": "1393512091276",
+ "create.time": "27 Feb 2014 14:41:31 GMT"
+ }
+ }
+}
+</pre></div></div>
+<div class="section">
+<h2>Deployment specification: <tt>app_configuration.json</tt><a name="Deployment_specification:_app_configuration.json"></a></h2>
+<p>This defines parameters that are to be used when creating the instance of the application, and instances of the individual components.</p>
+
+<div class="source">
+<pre>{
+ "schema": "http://example.org/specification/v2.0.0",
+
+ "global": {
+
+ "zookeeper.port": "2181",
+ "zookeeper.path": "/yarnapps_small_cluster",
+ "zookeeper.hosts": "zoo1,zoo2,zoo3",
+ "env.MALLOC_ARENA_MAX": "4",
+ "site.hbase.master.startup.retainassign": "true",
+ "site.fs.defaultFS": "hdfs://cluster:8020",
+ "site.fs.default.name": "hdfs://cluster:8020",
+ "site.hbase.master.info.port": "0",
+ "site.hbase.regionserver.info.port": "0"
+ },
+ "components": {
+
+ "worker": {
+ "jvm.heapsize": "512M"
+ },
+ "master": {
+ "jvm.heapsize": "512M"
+ }
+ }
+}
+</pre></div>
+<p>The resolved specification defines the values that are passed to the different components.</p>
+
+<div class="source">
+<pre>{
+ "schema": "http://example.org/specification/v2.0.0",
+
+ "global": {
+ "zookeeper.port": "2181",
+ "zookeeper.path": "/yarnapps_small_cluster",
+ "zookeeper.hosts": "zoo1,zoo2,zoo3",
+ "env.MALLOC_ARENA_MAX": "4",
+ "site.hbase.master.startup.retainassign": "true",
+ "site.fs.defaultFS": "hdfs://cluster:8020",
+ "site.fs.default.name": "hdfs://cluster:8020",
+ "site.hbase.master.info.port": "0",
+ "site.hbase.regionserver.info.port": "0"
+ },
+ "components": {
+
+ "worker": {
+ "zookeeper.port": "2181",
+ "zookeeper.path": "/yarnapps_small_cluster",
+ "zookeeper.hosts": "zoo1,zoo2,zoo3",
+ "env.MALLOC_ARENA_MAX": "4",
+ "site.hbase.master.startup.retainassign": "true",
+ "site.fs.defaultFS": "hdfs://cluster:8020",
+ "site.fs.default.name": "hdfs://cluster:8020",
+ "site.hbase.master.info.port": "0",
+ "site.hbase.regionserver.info.port": "0",
+ "jvm.heapsize": "512M"
+ },
+ "master": {
+ "zookeeper.port": "2181",
+ "zookeeper.path": "/yarnapps_small_cluster",
+ "zookeeper.hosts": "zoo1,zoo2,zoo3",
+ "env.MALLOC_ARENA_MAX": "4",
+ "site.hbase.master.startup.retainassign": "true",
+ "site.fs.defaultFS": "hdfs://cluster:8020",
+ "site.fs.default.name": "hdfs://cluster:8020",
+ "site.hbase.master.info.port": "0",
+ "site.hbase.regionserver.info.port": "0",
+ "jvm.heapsize": "512M"
+ }
+ }
+}
+</pre></div>
+<p>The <tt>site.</tt> properties have been passed down to each component, components whose templates may generate local site configurations. The override model does not prevent any component from overriding global configuration so as to create local configurations incompatible with the global state. (i.e., there is no way to declare an attribute as final). It is the responsibility of the author of the configuration file (and their tools) to detect such issues.</p></div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container">
+ <div class="row span12">Copyright © 2014
+ <a href="http://www.apache.org/">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+ <?xml version="1.0" encoding="UTF-8"?>
+<div class="row-fluid">Apache Slider, Slider, Apache, and the Apache Incubator logo are trademarks of The Apache Software Foundation.</div>
+
+ <p id="poweredBy" class="pull-right">
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../images/logos/maven-feather.png" />
+ </a>
+ </p>
+
+ </div>
+ </footer>
+ </body>
+</html>
Propchange: incubator/slider/site/content/configuration/core.html
------------------------------------------------------------------------------
svn:eol-style = native
Added: incubator/slider/site/content/configuration/index-markdown.html
URL: http://svn.apache.org/viewvc/incubator/slider/site/content/configuration/index-markdown.html?rev=1598665&view=auto
==============================================================================
--- incubator/slider/site/content/configuration/index-markdown.html (added)
+++ incubator/slider/site/content/configuration/index-markdown.html Fri May 30 16:47:13 2014
@@ -0,0 +1,202 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2014-05-30
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20140530" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache Slider 0.30 (incubating) - </title>
+ <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../css/site.css" />
+ <link rel="stylesheet" href="../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+ </head>
+ <body class="topBarEnabled">
+
+
+
+
+
+ <div id="topbar" class="navbar navbar-fixed-top ">
+ <div class="navbar-inner">
+ <div class="container"><div class="nav-collapse">
+
+
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Documentation <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li class="dropdown-submenu">
+ <a href="../project-info.html" title="Project Information">Project Information</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../index.html" title="About">About</a>
+</li>
+ <li> <a href="../plugin-management.html" title="Plugin Management">Plugin Management</a>
+</li>
+ <li> <a href="../distribution-management.html" title="Distribution Management">Distribution Management</a>
+</li>
+ <li> <a href="../dependency-info.html" title="Dependency Information">Dependency Information</a>
+</li>
+ <li> <a href="../dependency-convergence.html" title="Dependency Convergence">Dependency Convergence</a>
+</li>
+ <li> <a href="../source-repository.html" title="Source Repository">Source Repository</a>
+</li>
+ <li> <a href="../mail-lists.html" title="Mailing Lists">Mailing Lists</a>
+</li>
+ <li> <a href="../issue-tracking.html" title="Issue Tracking">Issue Tracking</a>
+</li>
+ <li> <a href="../integration.html" title="Continuous Integration">Continuous Integration</a>
+</li>
+ <li> <a href="../plugins.html" title="Project Plugins">Project Plugins</a>
+</li>
+ <li> <a href="../license.html" title="Project License">Project License</a>
+</li>
+ <li> <a href="../modules.html" title="Project Modules">Project Modules</a>
+</li>
+ <li> <a href="../dependency-management.html" title="Dependency Management">Dependency Management</a>
+</li>
+ <li> <a href="../team-list.html" title="Project Team">Project Team</a>
+</li>
+ <li> <a href="../project-summary.html" title="Project Summary">Project Summary</a>
+</li>
+ <li> <a href="../dependencies.html" title="Dependencies">Dependencies</a>
+</li>
+ </ul>
+ </li>
+
+ <li class="dropdown-submenu">
+ <a href="../project-reports.html" title="Project Reports">Project Reports</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../surefire-report.html" title="Surefire Report">Surefire Report</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documents <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../getting_started.html" title="Getting Started">Getting Started</a>
+</li>
+
+ <li> <a href="../manpage.html" title="manpage">manpage</a>
+</li>
+
+ <li> <a href="../troubleshooting.html" title="Troubleshooting">Troubleshooting</a>
+</li>
+
+ <li> <a href="../architecture/index.html" title="Architecture">Architecture</a>
+</li>
+
+ <li> <a href="../developing/index.html" title="Developing">Developing</a>
+</li>
+
+ <li> <a href="../exitcodes.html" title="Exitcodes">Exitcodes</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+ <div class="container">
+ <div id="banner">
+ <div class="pull-left">
+ <a href=".././" id="bannerLeft">
+ <h2>Apache Slider (incubating)</h2>
+ </a>
+ </div>
+ <div class="pull-right"> <div id="bannerRight">
+ <img src="http://incubator.apache.org/images/apache-incubator-logo.png" />
+ </div>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li id="publishDate">Last Published: 2014-05-30</li>
+
+
+
+ <li id="projectVersion" class="pull-right">Version: 0.30</li>
+
+ </ul>
+ </div>
+
+
+
+ <div id="bodyColumn" >
+
+ <!-- -
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. --><h1>Apache Slider: Specification of an application instance, revision 2.0</h1>
+<p>The specification of an applicaton, comprises 1. The persistent description of an application’s configuration 1. The persistent description of the desired topology and YARN resource requirements. 1. The dynamic description of the running application, including information on the location of components and aggregated statistics. </p>
+
+<ol style="list-style-type: decimal">
+
+<li><a href="redesign.md">Redesign</a></li>
+
+<li><a href="specification.md">Specification</a></li>
+
+<li><a href="original-hbase.json">Example: current</a></li>
+
+<li><a href="proposed-hbase.json">Example: proposed</a></li>
+</ol>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container">
+ <div class="row span12">Copyright © 2014
+ <a href="http://www.apache.org/">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+ <?xml version="1.0" encoding="UTF-8"?>
+<div class="row-fluid">Apache Slider, Slider, Apache, and the Apache Incubator logo are trademarks of The Apache Software Foundation.</div>
+
+ <p id="poweredBy" class="pull-right">
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../images/logos/maven-feather.png" />
+ </a>
+ </p>
+
+ </div>
+ </footer>
+ </body>
+</html>
Propchange: incubator/slider/site/content/configuration/index-markdown.html
------------------------------------------------------------------------------
svn:eol-style = native
Added: incubator/slider/site/content/configuration/index.html
URL: http://svn.apache.org/viewvc/incubator/slider/site/content/configuration/index.html?rev=1598665&view=auto
==============================================================================
--- incubator/slider/site/content/configuration/index.html (added)
+++ incubator/slider/site/content/configuration/index.html Fri May 30 16:47:13 2014
@@ -0,0 +1,208 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2014-05-30
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20140530" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache Slider 0.30 (incubating) - </title>
+ <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../css/site.css" />
+ <link rel="stylesheet" href="../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+ </head>
+ <body class="topBarEnabled">
+
+
+
+
+
+ <div id="topbar" class="navbar navbar-fixed-top ">
+ <div class="navbar-inner">
+ <div class="container"><div class="nav-collapse">
+
+
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Documentation <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li class="dropdown-submenu">
+ <a href="../project-info.html" title="Project Information">Project Information</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../index.html" title="About">About</a>
+</li>
+ <li> <a href="../plugin-management.html" title="Plugin Management">Plugin Management</a>
+</li>
+ <li> <a href="../distribution-management.html" title="Distribution Management">Distribution Management</a>
+</li>
+ <li> <a href="../dependency-info.html" title="Dependency Information">Dependency Information</a>
+</li>
+ <li> <a href="../dependency-convergence.html" title="Dependency Convergence">Dependency Convergence</a>
+</li>
+ <li> <a href="../source-repository.html" title="Source Repository">Source Repository</a>
+</li>
+ <li> <a href="../mail-lists.html" title="Mailing Lists">Mailing Lists</a>
+</li>
+ <li> <a href="../issue-tracking.html" title="Issue Tracking">Issue Tracking</a>
+</li>
+ <li> <a href="../integration.html" title="Continuous Integration">Continuous Integration</a>
+</li>
+ <li> <a href="../plugins.html" title="Project Plugins">Project Plugins</a>
+</li>
+ <li> <a href="../license.html" title="Project License">Project License</a>
+</li>
+ <li> <a href="../modules.html" title="Project Modules">Project Modules</a>
+</li>
+ <li> <a href="../dependency-management.html" title="Dependency Management">Dependency Management</a>
+</li>
+ <li> <a href="../team-list.html" title="Project Team">Project Team</a>
+</li>
+ <li> <a href="../project-summary.html" title="Project Summary">Project Summary</a>
+</li>
+ <li> <a href="../dependencies.html" title="Dependencies">Dependencies</a>
+</li>
+ </ul>
+ </li>
+
+ <li class="dropdown-submenu">
+ <a href="../project-reports.html" title="Project Reports">Project Reports</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../surefire-report.html" title="Surefire Report">Surefire Report</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documents <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../getting_started.html" title="Getting Started">Getting Started</a>
+</li>
+
+ <li> <a href="../manpage.html" title="manpage">manpage</a>
+</li>
+
+ <li> <a href="../troubleshooting.html" title="Troubleshooting">Troubleshooting</a>
+</li>
+
+ <li> <a href="../architecture/index.html" title="Architecture">Architecture</a>
+</li>
+
+ <li> <a href="../developing/index.html" title="Developing">Developing</a>
+</li>
+
+ <li> <a href="../exitcodes.html" title="Exitcodes">Exitcodes</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+ <div class="container">
+ <div id="banner">
+ <div class="pull-left">
+ <a href=".././" id="bannerLeft">
+ <h2>Apache Slider (incubating)</h2>
+ </a>
+ </div>
+ <div class="pull-right"> <div id="bannerRight">
+ <img src="http://incubator.apache.org/images/apache-incubator-logo.png" />
+ </div>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li id="publishDate">Last Published: 2014-05-30</li>
+
+
+
+ <li id="projectVersion" class="pull-right">Version: 0.30</li>
+
+ </ul>
+ </div>
+
+
+
+ <div id="bodyColumn" >
+
+ <!-- -
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. --><h1>Apache Slider: Specification of an application instance, revision 2.0</h1>
+<p>The specification of an applicaton, comprises 1. The persistent description of an application’s configuration 1. The persistent description of the desired topology and YARN resource requirements. 1. The dynamic description of the running application, including information on the location of components and aggregated statistics. </p>
+<p>The specifics of this are covered in the <a href="core.md">Core Configuration Specification</a></p>
+<div class="section">
+<h2>Historical References<a name="Historical_References"></a></h2>
+
+<ol style="list-style-type: decimal">
+
+<li><a href="specification.html">Specification</a></li>
+
+<li><a href="redesign.html">Redesign</a></li>
+</ol>
+
+<ol style="list-style-type: decimal">
+
+<li><a href="original-hbase.json">Example: current</a></li>
+
+<li><a href="proposed-hbase.json">Example: proposed</a></li>
+</ol></div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container">
+ <div class="row span12">Copyright © 2014
+ <a href="http://www.apache.org/">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+ <?xml version="1.0" encoding="UTF-8"?>
+<div class="row-fluid">Apache Slider, Slider, Apache, and the Apache Incubator logo are trademarks of The Apache Software Foundation.</div>
+
+ <p id="poweredBy" class="pull-right">
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../images/logos/maven-feather.png" />
+ </a>
+ </p>
+
+ </div>
+ </footer>
+ </body>
+</html>
Propchange: incubator/slider/site/content/configuration/index.html
------------------------------------------------------------------------------
svn:eol-style = native
Added: incubator/slider/site/content/configuration/redesign.html
URL: http://svn.apache.org/viewvc/incubator/slider/site/content/configuration/redesign.html?rev=1598665&view=auto
==============================================================================
--- incubator/slider/site/content/configuration/redesign.html (added)
+++ incubator/slider/site/content/configuration/redesign.html Fri May 30 16:47:13 2014
@@ -0,0 +1,572 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2014-05-30
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20140530" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache Slider 0.30 (incubating) - </title>
+ <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../css/site.css" />
+ <link rel="stylesheet" href="../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+ </head>
+ <body class="topBarEnabled">
+
+
+
+
+
+ <div id="topbar" class="navbar navbar-fixed-top ">
+ <div class="navbar-inner">
+ <div class="container"><div class="nav-collapse">
+
+
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Documentation <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li class="dropdown-submenu">
+ <a href="../project-info.html" title="Project Information">Project Information</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../index.html" title="About">About</a>
+</li>
+ <li> <a href="../plugin-management.html" title="Plugin Management">Plugin Management</a>
+</li>
+ <li> <a href="../distribution-management.html" title="Distribution Management">Distribution Management</a>
+</li>
+ <li> <a href="../dependency-info.html" title="Dependency Information">Dependency Information</a>
+</li>
+ <li> <a href="../dependency-convergence.html" title="Dependency Convergence">Dependency Convergence</a>
+</li>
+ <li> <a href="../source-repository.html" title="Source Repository">Source Repository</a>
+</li>
+ <li> <a href="../mail-lists.html" title="Mailing Lists">Mailing Lists</a>
+</li>
+ <li> <a href="../issue-tracking.html" title="Issue Tracking">Issue Tracking</a>
+</li>
+ <li> <a href="../integration.html" title="Continuous Integration">Continuous Integration</a>
+</li>
+ <li> <a href="../plugins.html" title="Project Plugins">Project Plugins</a>
+</li>
+ <li> <a href="../license.html" title="Project License">Project License</a>
+</li>
+ <li> <a href="../modules.html" title="Project Modules">Project Modules</a>
+</li>
+ <li> <a href="../dependency-management.html" title="Dependency Management">Dependency Management</a>
+</li>
+ <li> <a href="../team-list.html" title="Project Team">Project Team</a>
+</li>
+ <li> <a href="../project-summary.html" title="Project Summary">Project Summary</a>
+</li>
+ <li> <a href="../dependencies.html" title="Dependencies">Dependencies</a>
+</li>
+ </ul>
+ </li>
+
+ <li class="dropdown-submenu">
+ <a href="../project-reports.html" title="Project Reports">Project Reports</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../surefire-report.html" title="Surefire Report">Surefire Report</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documents <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../getting_started.html" title="Getting Started">Getting Started</a>
+</li>
+
+ <li> <a href="../manpage.html" title="manpage">manpage</a>
+</li>
+
+ <li> <a href="../troubleshooting.html" title="Troubleshooting">Troubleshooting</a>
+</li>
+
+ <li> <a href="../architecture/index.html" title="Architecture">Architecture</a>
+</li>
+
+ <li> <a href="../developing/index.html" title="Developing">Developing</a>
+</li>
+
+ <li> <a href="../exitcodes.html" title="Exitcodes">Exitcodes</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+ <div class="container">
+ <div id="banner">
+ <div class="pull-left">
+ <a href=".././" id="bannerLeft">
+ <h2>Apache Slider (incubating)</h2>
+ </a>
+ </div>
+ <div class="pull-right"> <div id="bannerRight">
+ <img src="http://incubator.apache.org/images/apache-incubator-logo.png" />
+ </div>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li id="publishDate">Last Published: 2014-05-30</li>
+
+
+
+ <li id="projectVersion" class="pull-right">Version: 0.30</li>
+
+ </ul>
+ </div>
+
+
+
+ <div id="bodyColumn" >
+
+ <!-- -
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. --><h1>Apache Slider Cluster Specification</h1>
+<div class="section">
+<div class="section">
+<h3>Notation:<a name="Notation:"></a></h3>
+<p>In this document, a full path to a value is represented as a path <tt>options/zookeeper.port</tt> ; an assigment as <tt>options/zookeeper.port=2181</tt>.</p>
+<p>A wildcard indicates all entries matching a path: <tt>options/zookeeper.*</tt> or <tt>/roles/*/yarn.memory</tt></p></div></div>
+<div class="section">
+<h2>History<a name="History"></a></h2>
+<p>The Slider cluster specification was implicitly defined in the file <tt>org.apache.slider.api.ClusterDescription</tt>. It had a number of roles</p>
+
+<ol style="list-style-type: decimal">
+
+<li>Persistent representaton of cluster state</li>
+
+<li>Internal model of desired cluster state within the Application Master.</li>
+
+<li>Dynamic representation of current cluster state when the AM was queried, marshalled over the network as JSON.</li>
+
+<li>Description of updated state when reconfiguring a running cluster.</li>
+</ol>
+<p>Initially the dynamic status included a complete history of all containers -this soon highlit some restrictions on the maximum size of a JSON-formatted string in Hadoop’s “classic” RPC: 32K, after which the string was silently truncated. Accordingly, this history was dropped.</p>
+<p>Having moved to Protocol Buffers as the IPC wire format, with a web view alongside, this history could be reconsidered.</p>
+<p>The initial design place most values into the root entry, and relied on Jaxon introspection to set and retrieve the values -it was a Java-first specification, with no external specificatin or regression tests.</p>
+<p>As the number of entries in the root increased, the design switched to storing more attributes into specific sections <i>under</i> the root path:</p>
+
+<ul>
+
+<li><tt>info</tt>: read-only information about the cluster.</li>
+
+<li><tt>statistics</tt>: Numeric statistics about the cluster</li>
+</ul>
+<h1>Sections</h1></div>
+<div class="section">
+<h2>Root<a name="Root"></a></h2>
+<p>Contains various string and integer values</p>
+
+<div class="source">
+<pre>"version": "1.0",
+"name": "test_cluster_lifecycle",
+"type": "hbase",
+"state": 3,
+"createTime": 1393512091276,
+"updateTime": 1393512117286,
+"originConfigurationPath": "hdfs://sandbox:8020/user/stevel/.slider/cluster/test_cluster_lifecycle/snapshot",
+"generatedConfigurationPath": "hdfs://sandbox:8020/user/stevel/.slider/cluster/test_cluster_lifecycle/generated",
+"dataPath": "hdfs://sandbox:8020/user/stevel/.slider/cluster/test_cluster_lifecycle/database",
+</pre></div>
+
+<ul>
+
+<li>
+<p><tt>version</tt>: version of the JSON file. Not currently used to validate version compatibility; at this point in time releases may not be able to read existing .json files.</p></li>
+
+<li>
+<p><tt>name</tt>: cluster name</p></li>
+
+<li><tt>type</tt>: reference to the provider type -this triggers a Hadoop configuration property lookup to find the implementation classes.</li>
+
+<li>
+<p><tt>state</tt>: an enumeration value of the cluster state.</p>
+
+<div class="source">
+<pre>int STATE_INCOMPLETE = 0;
+int STATE_SUBMITTED = 1;
+int STATE_CREATED = 2;
+int STATE_LIVE = 3;
+int STATE_STOPPED = 4;
+int STATE_DESTROYED = 5;
+</pre></div></li>
+</ul>
+<p>Only two states are persisted, “incomplete” and “created”, though more are used internally. The <tt>incomplete</tt> state is used during cluster create/build, allowing an incomplete JSON file to be written -so minimising the window for race conditions on cluster construction.</p>
+
+<ul>
+
+<li><tt>createTime</tt> and <tt>updateTime</tt>: timestamps, informative only. The <tt>createTime</tt> value is duplicated in <tt>/info/createTimeMillis</tt></li>
+
+<li><tt>originConfigurationPath</tt>, <tt>generatedConfigurationPath</tt>, <tt>dataPath</tt> paths used internally -if changed the cluster may not start.</li>
+</ul>
+<p><i>Proposed</i>: 1. Move all state bar <tt>name</tt> and cluster state into a section <tt>/slider-internal</tt>. 1. The cluster state is moved from an enum to a simple boolean, <tt>valid</tt>, set to true when the cluster JSON has been fully constructed.</p></div>
+<div class="section">
+<h2><tt>/info</tt><a name="ainfo"></a></h2>
+<p>Read-only list of information about the application. Generally this is intended to be used for debugging and testing.</p>
+<div class="section">
+<h3>Persisted values: static information about the file history<a name="Persisted_values:_static_information_about_the_file_history"></a></h3>
+
+<div class="source">
+<pre>"info" : {
+ "create.hadoop.deployed.info" : "(detached from release-2.3.0) @dfe46336fbc6a044bc124392ec06b85",
+ "create.application.build.info" : "Slider Core-0.13.0-SNAPSHOT Built against commit# 1a94ee4aa1 on Java 1.7.0_45 by stevel",
+ "create.hadoop.build.info" : "2.3.0",
+ "create.time.millis" : "1393512091276",
+},
+</pre></div>
+<p><i>Proposed</i>: move persisted info K-V pairs to a section <tt>/diagnostics</tt>.</p></div>
+<div class="section">
+<h3>Dynamic values:<a name="Dynamic_values:"></a></h3>
+<p>whether the AM supports service restart without killing all the containers hosting the role instances:</p>
+
+<div class="source">
+<pre>"slider.am.restart.supported" : "false",
+</pre></div>
+<p>timestamps of the cluster going live, and when the status query was made</p>
+
+<div class="source">
+<pre>"live.time" : "27 Feb 2014 14:41:56 GMT",
+"live.time.millis" : "1393512116881",
+"status.time" : "27 Feb 2014 14:42:08 GMT",
+"status.time.millis" : "1393512128726",
+</pre></div>
+<p>yarn data provided to the AM</p>
+
+<div class="source">
+<pre>"yarn.vcores" : "32",
+"yarn.memory" : "2048",
+</pre></div>
+<p>information about the application and hadoop versions in use. Here the application was built using Hadoop 2.3.0, but is running against the version of Hadoop built for HDP-2.</p>
+
+<div class="source">
+<pre>"status.application.build.info" : "Slider Core-0.13.0-SNAPSHOT Built against commit# 1a94ee4aa1 on Java 1.7.0_45 by stevel",
+"status.hadoop.build.info" : "2.3.0",
+"status.hadoop.deployed.info" : "bigwheel-m16-2.2.0 @704f1e463ebc4fb89353011407e965"
+</pre></div>
+<p>## <tt>instances</tt></p>
+<p>Information about the live containers in a cluster</p>
+
+<div class="source">
+<pre> "instances": {
+ "slider": [ "container_1393511571284_0002_01_000001" ],
+ "master": [ "container_1393511571284_0002_01_000003" ],
+ "worker": [
+ "container_1393511571284_0002_01_000002",
+ "container_1393511571284_0002_01_000004"
+ ]
+ },
+</pre></div>
+<p>There’s no information about location, nor is there any history about containers that are no longer part of the cluster (i.e. failed & released containers). </p>
+<p>It could be possible to include a list of previous containers, though Slider would need to be selective about how many to store (or how much detail to retain) on those previous containers.</p>
+<p>Perhaps the list could be allowed to grow without limit, but detail only preserved on the last 100. If more containers fail than that, there is likely to be a problem which the most recent containers will also display.</p>
+<p><i>Proposed</i> </p>
+
+<ol style="list-style-type: decimal">
+
+<li>Return to the full serialization of container state -but only for running containers.</li>
+
+<li>Have a list of failed containers, but only include last 8; make it a rolling buffer. This avoids a significantly failing role to overload the status document.</li>
+</ol>
+<p>## <tt>statistics</tt></p>
+<p>Statistics on each role. </p>
+<p>They can be divided into counters that only increase</p>
+
+<div class="source">
+<pre>"containers.start.completed": 0,
+"containers.start.failed": 0,
+"containers.failed": 0,
+"containers.completed": 0,
+"containers.requested": 0
+</pre></div>
+<p>and those that vary depending upon the current state</p>
+
+<div class="source">
+<pre>"containers.live": 0,
+"containers.active.requests": 0,
+"containers.desired": 0,
+</pre></div>
+
+<ul>
+
+<li>Propose: move these values out of statistics into some other section, as they are state, not statistics*</li>
+</ul>
+
+<div class="source">
+<pre> "statistics": {
+ "worker": {
+ "containers.start.completed": 0,
+ "containers.live": 2,
+ "containers.start.failed": 0,
+ "containers.active.requests": 0,
+ "containers.failed": 0,
+ "containers.completed": 0,
+ "containers.desired": 2,
+ "containers.requested": 0
+ },
+ "slider": {
+ "containers.unknown.completed": 0,
+ "containers.start.completed": 3,
+ "containers.live": 1,
+ "containers.start.failed": 0,
+ "containers.failed": 0,
+ "containers.completed": 0,
+ "containers.surplus": 0
+ },
+ "master": {
+ "containers.start.completed": 0,
+ "containers.live": 1,
+ "containers.start.failed": 0,
+ "containers.active.requests": 0,
+ "containers.failed": 0,
+ "containers.completed": 0,
+ "containers.desired": 1,
+ "containers.requested": 0
+ }
+ },
+</pre></div>
+<p>The <tt>/statistics/slider</tt> section is unusual in that it provides the aggregate statistics of the cluster -this is not obvious. A different name could be used -but again, there’s a risk of clash with or confusion with a role. </p>
+<p>Better to have a specific <tt>/statistics/cluster</tt> element, and to move the roles’ statistics under <tt>/statistics/roles</tt>:</p>
+
+<div class="source">
+<pre>"statistics": {
+ "cluster": {
+ "containers.unknown.completed": 0,
+ "containers.start.completed": 3,
+ "containers.live": 1,
+ "containers.start.failed": 0,
+ "containers.failed": 0,
+ "containers.completed": 0,
+ "containers.surplus": 0
+
+ },
+ "roles": {
+ "worker": {
+ "containers.start.completed": 0,
+ "containers.live": 2,
+ "containers.start.failed": 0,
+ "containers.active.requests": 0,
+ "containers.failed": 0,
+ "containers.completed": 0,
+ "containers.desired": 2,
+ "containers.requested": 0
+ },
+ "master": {
+ "containers.start.completed": 0,
+ "containers.live": 1,
+ "containers.start.failed": 0,
+ "containers.active.requests": 0,
+ "containers.failed": 0,
+ "containers.completed": 0,
+ "containers.desired": 1,
+ "containers.requested": 0
+ }
+ }
+},
+</pre></div>
+<p>This approach allows extra statistics sections to be added (perhaps by providers), without any changes to the toplevel section.</p></div></div>
+<div class="section">
+<h2>Options<a name="Options"></a></h2>
+<p>A list of options used by Slider and its providers to build up the AM and the configurations of the deployed service components</p>
+
+<div class="source">
+<pre>"options": {
+ "zookeeper.port": "2181",
+ "site.hbase.master.startup.retainassign": "true",
+ "slider.cluster.application.image.path": "hdfs://sandbox:8020/hbase.tar.gz",
+ "site.fs.defaultFS": "hdfs://sandbox:8020",
+ "slider.container.failure.threshold": "5",
+ "site.fs.default.name": "hdfs://sandbox:8020",
+ "slider.cluster.directory.permissions": "0770",
+ "slider.am.monitoring.enabled": "false",
+ "zookeeper.path": "/yarnapps_slider_stevel_test_cluster_lifecycle",
+ "slider.tmp.dir": "hdfs://sandbox:8020/user/stevel/.slider/cluster/test_cluster_lifecycle/tmp/am",
+ "slider.data.directory.permissions": "0770",
+ "zookeeper.hosts": "sandbox",
+ "slider.container.failure.shortlife": "60"
+},
+</pre></div>
+<p>Some for these options have been created by slider itself (“slider.tmp.dir”) for internal use -and are cluster specific. If/when the ability to use an existing json file as a template for a new cluster is added, having these options in the configuration will create problems</p>
+<h1>Proposed Changes</h1></div>
+<div class="section">
+<h2>Move Slider internal state to <tt>/slider-internal</tt><a name="Move_Slider_internal_state_to_slider-internal"></a></h2>
+<p>Move all slider “private” data to an internal section,<tt>/slider-internal</tt> including those in the toplevel directory and in <tt>/options</tt></p></div>
+<div class="section">
+<h2>Allow <tt>/options</tt> and <tt>roles/*/</tt> options entries to take the value “null”.<a name="Allow_options_and_roles_options_entries_to_take_the_value_null."></a></h2>
+<p>This would be a definition that the value must be defined before the cluster can start. Provider templates could declare this.</p></div>
+<div class="section">
+<h2>Make client configuration retrieval hierarchical -and maybe move out of the<a name="Make_client_configuration_retrieval_hierarchical_-and_maybe_move_out_of_the"></a></h2>
+<p>status</p>
+<p>The current design assumes that it is a -site.xml file being served up. This does not work for alternate file formats generated by the Provider.</p></div>
+<div class="section">
+<h2>Role Options<a name="Role_Options"></a></h2>
+<p>The <tt>/roles/$ROLENAME/</tt> clauses each provide options for a specific role.</p>
+<p>This includes 1. <tt>role.instances</tt>: defines the number of instances of a role to create 1. <tt>env.</tt> environment variables for launching the container 1. <tt>yarn.</tt> properties to configure YARN requests. 1. <tt>jvm.heapsize</tt>: an option supported by some providers to fix the heap size of a component. 1. <tt>app.infoport</tt>: an option supported by some providers (e.g. HBase) to fix the port to which a role (master or worker) binds its web UI.</p>
+
+<div class="source">
+<pre> "worker": {
+ "yarn.memory": "768",
+ "env.MALLOC_ARENA_MAX": "4",
+ "role.instances": "0",
+ "role.name": "worker",
+ "jvm.heapsize": "512M",
+ "yarn.vcores": "1",
+ "app.infoport": "0"
+ },
+</pre></div>
+<p>In a live cluster, the role information also includes status information about the cluster.</p>
+
+<div class="source">
+<pre> "master": {
+ "yarn.memory": "1024",
+ "env.MALLOC_ARENA_MAX": "4",
+ "role.instances": "0",
+ "role.requested.instances": "0",
+ "role.name": "master",
+ "role.failed.starting.instances": "0",
+ "role.actual.instances": "0",
+ "jvm.heapsize": "512M",
+ "yarn.vcores": "1",
+ "role.releasing.instances": "0",
+ "role.failed.instances": "0",
+ "app.infoport": "0"
+ }
+</pre></div>
+<p>The role <tt>slider</tt> represents the Slider Application Master itself.</p>
+
+<div class="source">
+<pre> "slider": {
+ "yarn.memory": "256",
+ "env.MALLOC_ARENA_MAX": "4",
+ "role.instances": "1",
+ "role.name": "slider",
+ "jvm.heapsize": "256M",
+ "yarn.vcores": "1",
+ },
+</pre></div>
+<div class="section">
+<h3>Proposed:<a name="Proposed:"></a></h3>
+
+<ol style="list-style-type: decimal">
+
+<li>move all dynamic role status to its own clauses.</li>
+
+<li>use a simple inheritance model from <tt>/options</tt></li>
+
+<li>don’t allow role entries to alter the cluster state.</li>
+</ol></div>
+<div class="section">
+<h3>Proposed: <tt>/clientProperties</tt> continues return Key-val pairs<a name="Proposed:_clientProperties_continues_return_Key-val_pairs"></a></h3>
+<p>The <tt>/clientProperties</tt> section will remain, with key-val pairs of type string, the expectation being this is where providers can insert specific single attributes for client applications.</p>
+<p>These values can be converted to application-specific files on the client, in code -as done today in the Slider CLI-, or via template expansion (beyond the scope of this document.</p></div>
+<div class="section">
+<h3>Proposed: alongside <tt>/clientProperties</tt> comes <tt>/clientfiles</tt><a name="Proposed:_alongside_clientProperties_comes_clientfiles"></a></h3>
+<p>This section will list all files that an application instance can generate for clients, along with with a description.</p>
+
+<div class="source">
+<pre>"/clientfiles/hbase-site.xml": "site information for HBase"
+"/clientfiles/log4.properties": "log4.property file"
+</pre></div>
+<p>A new CLI command would be added to retrieve a client file. 1. The specific file must be named. 1. If it is not present, an error must be raised. 1. If it is present, it is downloaded and output to the console/to a named destination file/directory <tt>--outfile <file></tt> and <tt>--outdir <dir></tt> 1. If the <tt>--list</tt> argument is provided, the list of available files is returned (e.g.) </p>
+
+<div class="source">
+<pre>hbase-site.xml: site information for HBase
+log4.properties: log4.property file
+</pre></div>
+<p><i>No attempt to parse/process the body of the messages will be returned.</i></p>
+<p>In a REST implementation of the client API, /clientconf would be a path to the list of options; each file a path underneath.</p>
+<p>Client configuration file retrieval outside the status completely; the status just lists the possible values; a separate call returns them.</p>
+<p>This will permit binary content to be retrieved, and avoid any marshalling problems and inefficiencies.</p>
+<p>With this change, there will now be two ways to generate client configuration files</p>
+
+<ul>
+
+<li>Client-side: as today</li>
+
+<li>Server-side: via the provider</li>
+</ul>
+<p>Client side is more extensible as it allows for arbitrary clients; server-side is restricted to those files which the application provider is capable of generating. The advantage of the server-side option is that for those files about which the provider is aware of, they will be visible through the REST and Web UIs, so trivially retrieved.</p></div>
+<div class="section">
+<h3>Stop intermixing role specification with role current state<a name="Stop_intermixing_role_specification_with_role_current_state"></a></h3>
+<p>Create a new section, <tt>rolestatus</tt>, which lists the current status of the roles: how many are running vs requested, how many are being released.</p>
+<p>There’s some overlap here with the <tt>/statistics</tt> field, so we should either merge them or clearly separate the two. Only the <tt>role.failed</tt> properties match entries in the statistics -perhaps they should be cut.</p>
+<div class="section">
+<h4>provider-specific status<a name="provider-specific_status"></a></h4>
+<p>Allow providers to publish information to the status, in their own section.</p>
+<p>There already is support for providers updating the cluster status in Slider 12.1 and earlier, but it has flaws</p>
+<p>A key one is that it is done sychronously on a <tt>getStatus()</tt> call; as providers may perform a live query of their status (example, the HBase provider looks up the Web UI ports published by HBase to zookeeper), there’s overhead, and if the operation blocks (example: when HBase hasn’t ever been deployed and the zookeeper path is empty), then the status call blocks.</p>
+<p><i>Proposed:</i></p>
+
+<ol style="list-style-type: decimal">
+
+<li>There is a specific <tt>/provider</tt> section</li>
+
+<li>There’s no restriction on what JSON is permitted in this section.</li>
+
+<li>Providers may make their own updates to the application state to read and write this block -operations that are asynchronous to any status queries.</li>
+</ol></div></div></div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container">
+ <div class="row span12">Copyright © 2014
+ <a href="http://www.apache.org/">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+ <?xml version="1.0" encoding="UTF-8"?>
+<div class="row-fluid">Apache Slider, Slider, Apache, and the Apache Incubator logo are trademarks of The Apache Software Foundation.</div>
+
+ <p id="poweredBy" class="pull-right">
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../images/logos/maven-feather.png" />
+ </a>
+ </p>
+
+ </div>
+ </footer>
+ </body>
+</html>
Propchange: incubator/slider/site/content/configuration/redesign.html
------------------------------------------------------------------------------
svn:eol-style = native