You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@slider.apache.org by bi...@apache.org on 2014/07/07 21:03:40 UTC
svn commit: r1608561 [10/16] - in /incubator/slider/site/content: ./ css/
design/ design/registry/ design/specification/ developing/ docs/
docs/configuration/ docs/slider_specs/ downloads/ fonts/ images/ js/
release_notes/
Added: incubator/slider/site/content/docs/configuration/index.html
URL: http://svn.apache.org/viewvc/incubator/slider/site/content/docs/configuration/index.html?rev=1608561&view=auto
==============================================================================
--- incubator/slider/site/content/docs/configuration/index.html (added)
+++ incubator/slider/site/content/docs/configuration/index.html Mon Jul 7 19:03:38 2014
@@ -0,0 +1,217 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE- 2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <link href="/css/bootstrap.min.css" rel="stylesheet">
+ <link href="/css/bootstrap-theme.min.css" rel="stylesheet">
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
+ <script src="https://oss.maxcdn.com/libs/respond.js/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <link href="/css/slider.css" rel="stylesheet" type="text/css">
+ <link href="//netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
+ <title></title>
+
+</head>
+
+<body>
+<nav class="navbar navbar-default navbar-fixed-top" role="navigation">
+<div class="container-fluid">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#navbar-items">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a class="navbar-brand" href="/index.html">Slider</a>
+ </div>
+ <div class="collapse navbar-collapse" id="navbar-items">
+ <ul class="nav navbar-nav">
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">
+ Project <span class="caret"></span>
+ </a>
+<ul class="dropdown-menu">
+<li id="nav_index"><a href="/index.html">Home</a></li>
+<li id="nav_downloads"><a href="/downloads">Downloads</a></li>
+<li id="nav_source"><a href="https://git-wip-us.apache.org/repos/asf/incubator-slider.git"><i class="fa fa-external-link"></i> Source</a></li>
+<li><a href="http://www.apache.org/licenses/LICENSE-2.0"><i class="fa fa-external-link"></i> License</a></li>
+<li class="divider"></li>
+<li class="dropdown-header">Community</li>
+<li id="nav_team"><a href="/team.html">Team</a></li>
+<li id="nav_mailinglists"><a href="/mailing_lists.html">Mailing Lists</a></li>
+<li class="divider"></li>
+<li class="dropdown-header">Tools</li>
+<li id="nav_jira"><a href="https://issues.apache.org/jira/browse/SLIDER"><i class="fa fa-external-link"></i> Issues</a></li>
+<li id="nav_builds"><a href="https://builds.apache.org/job/Slider-develop"><i class="fa fa-external-link"></i> Builds</a></li>
+</ul>
+</li>
+
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="/design">
+ Design <span class="caret"></span>
+ </a>
+<ul class="dropdown-menu">
+<li id="nav_design_overview"><a href="/design/architecture.html">Overview</a></li>
+<li id="nav_app_needs"><a href="/docs/slider_specs/application_needs.html">Application Needs</a></li>
+<li id="nav_spec"><a href="/design/specification/index.html">Expected Behavior</a></li>
+<li id="nav_registry"><a href="/design/registry/index.html">Service Registry</a></li>
+<li id="nav_role_history"><a href="/design/rolehistory.html">Role History</a></li>
+</ul>
+</li>
+
+
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="/developing">
+ Developing <span class="caret"></span>
+ </a>
+<ul class="dropdown-menu">
+<li id="nav_dev_building"><a href="/developing/building.html">Building</a></li>
+<li id="nav_dev_debugging"><a href="/docs/debugging.html">Debugging</a></li>
+<li id="nav_dev_testing"><a href="/developing/testing.html">Testing</a></li>
+<li id="nav_dev_funtesting"><a href="/developing/functional_tests.html">Functional Testing</a></li>
+<li id="nav_dev_manual_testing"><a href="/developing/manual_testing.html">Manual Testing</a></li>
+<li id="nav_releasing"><a href="/developing/releasing.html">Releasing</a></li>
+</ul>
+</li>
+
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">
+ Using <span class="caret"></span>
+ </a>
+<ul class="dropdown-menu">
+<li id="nav_use_getting_started"><a href="/docs/getting_started.html">Getting Started</a></li>
+<li id="nav_use_manpage"><a href="/docs/manpage.html">Man Page</a></li>
+<li id="nav_use_examples"><a href="/docs/examples.html">Examples</a></li>
+<li id="nav_use_client_config"><a href="/docs/client-configuration.html">Client Configuration</a></li>
+<li id="nav_use_exitcodes"><a href="/docs/exitcodes.html">Client Exit Codes</a></li>
+<li id="nav_use_security"><a href="/docs/security.html">Security</a></li>
+<li id="nav_use_app_package"><a href="/docs/slider_specs/index.html">App Packaging</a></li>
+<li id="nav_use_app_configuration"><a href="/docs/configuration/index.html">App Configuration Model</a></li>
+<li id="nav_use_debugging"><a href="/docs/debugging.html">Debugging</a></li>
+<li id="nav_use_troubleshooting"><a href="/docs/troubleshooting.html">Troubleshooting</a></li>
+</ul>
+</li>
+
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">
+ ASF Links <span class="caret"></span>
+ </a>
+<ul class="dropdown-menu">
+<li><a href="http://www.apache.org"><i class="fa fa-external-link"></i> Apache Software Foundation</a></li>
+<li><a href="http://www.apache.org/foundation/sponsorship.html"><i class="fa fa-external-link"></i> Sponsorship</a></li>
+<li><a href="http://www.apache.org/security/"><i class="fa fa-external-link"></i> Security</a></li>
+<li><a href="http://www.apache.org/foundation/thanks.html"><i class="fa fa-external-link"></i> Thanks</a></li>
+</ul>
+</li>
+</ul>
+
+ </div>
+
+</div>
+</nav>
+
+
+<div class="container-fluid">
+<div class="row">
+ <div class="col-md-2" id="sidebar">
+ <div style="text-align: center">
+ <h1><a href="/index.html">Apache Slider (incubating)</a></h1>
+ <hr>
+Latest release: <strong>0.30-incubating</strong><br>
+ <br>
+ <a id="download-button-sidebar" class="btn btn-success btn-block" href="/downloads/" role="button">Download</a>
+ </div>
+ <hr>
+ <a href="http://incubator.apache.org"><img id="logo" alt="Apache Incubator ™" class="img-responsive" src="http://incubator.apache.org/images/apache-incubator-logo.png"/></a>
+ </div>
+ <div class="col-md-8 col-md-offset-1">
+
+ <div id="content">
+
+ <h1 class="title"></h1>
+
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<h1 id="apache-slider-specification-of-an-application-instance-revision-20">Apache Slider: Specification of an application instance, revision 2.0</h1>
+<p>The specification of an applicaton, comprises
+1. The persistent description of an application's configuration
+1. The persistent description of the desired topology and YARN resource
+requirements.
+1. The dynamic description of the running application, including information
+on the location of components and aggregated statistics. </p>
+<p>The specifics of this are covered in the <a href="core.html">Core Configuration Specification</a></p>
+<h2 id="historical-references">Historical References</h2>
+<ol>
+<li><a href="specification.html">Specification</a></li>
+<li>
+<p><a href="redesign.html">Redesign</a></p>
+</li>
+<li>
+<p><a href="original-hbase.json">Example: current</a></p>
+</li>
+<li><a href="proposed-hbase.json">Example: proposed</a></li>
+</ol>
+ </div>
+
+ <div id="footer">
+ <a alt="Apache Software Foundation" href="http://www.apache.org">
+ <img id="asf-logo" alt="Apache Software Foundation" src="/images/feather-small.gif" width="100">
+ </a>
+ <div class="copyright">
+ <p>
+ Copyright © 2011-2014 The Apache Software Foundation, Licensed under
+ the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.
+ Apache Slider, Slider, Apache, the Apache feather logo, and the Apache Incubator
+ project logo are trademarks of the <a href="http://www.apache.org">Apache Software Foundation</a>.<br />
+ Site created with <a href="http://getbootstrap.com/">Bootstrap</a> including icons from <a href="http://glyphicons.com/">GLYPHICONS</a> and <a href="http://fontawesome.io/">Font Awesome</a>.
+ </p>
+ </div>
+ </div>
+ </div>
+
+</div>
+</div>
+
+<script type="text/javascript">
+
+</script>
+</body>
+</html>
Propchange: incubator/slider/site/content/docs/configuration/index.html
------------------------------------------------------------------------------
svn:eol-style = native
Added: incubator/slider/site/content/docs/configuration/original-hbase.json
URL: http://svn.apache.org/viewvc/incubator/slider/site/content/docs/configuration/original-hbase.json?rev=1608561&view=auto
==============================================================================
--- incubator/slider/site/content/docs/configuration/original-hbase.json (added)
+++ incubator/slider/site/content/docs/configuration/original-hbase.json Mon Jul 7 19:03:38 2014
@@ -0,0 +1,139 @@
+{
+ "version": "1.0",
+ "name": "test_cluster_lifecycle",
+ "type": "hbase",
+ "state": 3,
+ "createTime": 1393512091276,
+ "updateTime": 1393512117286,
+ "originConfigurationPath": "hdfs://sandbox:8020/user/stevel/.slider/cluster/test_cluster_lifecycle/snapshot",
+ "generatedConfigurationPath": "hdfs://sandbox:8020/user/stevel/.slider/cluster/test_cluster_lifecycle/generated",
+ "dataPath": "hdfs://sandbox:8020/user/stevel/.slider/cluster/test_cluster_lifecycle/database",
+ "options": {
+ "zookeeper.port": "2181",
+ "site.hbase.master.startup.retainassign": "true",
+ "slider.cluster.application.image.path": "hdfs://sandbox:8020/hbase.tar.gz",
+ "site.fs.defaultFS": "hdfs://sandbox:8020",
+ "slider.container.failure.threshold": "5",
+ "site.fs.default.name": "hdfs://sandbox:8020",
+ "slider.cluster.directory.permissions": "0770",
+ "slider.am.monitoring.enabled": "false",
+ "zookeeper.path": "/yarnapps_slider_stevel_test_cluster_lifecycle",
+ "slider.tmp.dir": "hdfs://sandbox:8020/user/stevel/.slider/cluster/test_cluster_lifecycle/tmp/am",
+ "slider.data.directory.permissions": "0770",
+ "zookeeper.hosts": "sandbox",
+ "slider.container.failure.shortlife": "60"
+ },
+ "info": {
+ "create.hadoop.deployed.info": "(detached from release-2.3.0) @dfe46336fbc6a044bc124392ec06b85",
+ "create.application.build.info": "Slider Core-0.13.0-SNAPSHOT Built against commit# 1a94ee4aa1 on Java 1.7.0_45 by stevel",
+ "create.hadoop.build.info": "2.3.0",
+ "create.time.millis": "1393512091276",
+ "create.time": "27 Feb 2014 14:41:31 GMT",
+ "slider.am.restart.supported": "false",
+ "live.time": "27 Feb 2014 14:41:56 GMT",
+ "live.time.millis": "1393512116881",
+ "status.time": "27 Feb 2014 14:42:08 GMT",
+ "status.time.millis": "1393512128726",
+ "yarn.vcores": "32",
+ "yarn.memory": "2048",
+ "status.application.build.info": "Slider Core-0.13.0-SNAPSHOT Built against commit# 1a94ee4aa1 on Java 1.7.0_45 by stevel",
+ "status.hadoop.build.info": "2.3.0",
+ "status.hadoop.deployed.info": "bigwheel-m16-2.2.0 @704f1e463ebc4fb89353011407e965"
+ },
+ "statistics": {
+ "worker": {
+ "containers.start.started": 0,
+ "containers.live": 0,
+ "containers.start.failed": 0,
+ "containers.active.requests": 0,
+ "containers.failed": 0,
+ "containers.completed": 0,
+ "containers.desired": 0,
+ "containers.requested": 0
+ },
+ "slider": {
+ "containers.unknown.completed": 0,
+ "containers.start.started": 0,
+ "containers.live": 1,
+ "containers.start.failed": 0,
+ "containers.failed": 0,
+ "containers.completed": 0,
+ "containers.surplus": 0
+ },
+ "master": {
+ "containers.start.started": 0,
+ "containers.live": 0,
+ "containers.start.failed": 0,
+ "containers.active.requests": 0,
+ "containers.failed": 0,
+ "containers.completed": 0,
+ "containers.desired": 0,
+ "containers.requested": 0
+ }
+ },
+ "status": {
+ },
+ "instances": {
+ "slider": [ "container_1393511571284_0002_01_000001" ]
+ },
+ "roles": {
+ "worker": {
+ "yarn.memory": "768",
+ "env.MALLOC_ARENA_MAX": "4",
+ "role.instances": "0",
+ "role.requested.instances": "0",
+ "role.name": "worker",
+ "role.failed.starting.instances": "0",
+ "role.actual.instances": "0",
+ "jvm.heapsize": "512M",
+ "yarn.vcores": "1",
+ "role.releasing.instances": "0",
+ "role.failed.instances": "0",
+ "app.infoport": "0"
+ },
+ "slider": {
+ "yarn.memory": "256",
+ "env.MALLOC_ARENA_MAX": "4",
+ "role.instances": "1",
+ "role.requested.instances": "0",
+ "role.name": "slider",
+ "role.failed.starting.instances": "0",
+ "role.actual.instances": "1",
+ "jvm.heapsize": "256M",
+ "yarn.vcores": "1",
+ "role.releasing.instances": "0",
+ "role.failed.instances": "0"
+ },
+ "master": {
+ "yarn.memory": "1024",
+ "env.MALLOC_ARENA_MAX": "4",
+ "role.instances": "0",
+ "role.requested.instances": "0",
+ "role.name": "master",
+ "role.failed.starting.instances": "0",
+ "role.actual.instances": "0",
+ "jvm.heapsize": "512M",
+ "yarn.vcores": "1",
+ "role.releasing.instances": "0",
+ "role.failed.instances": "0",
+ "app.infoport": "0"
+ }
+ },
+ "clientProperties": {
+ "fs.defaultFS": "hdfs://sandbox:8020",
+ "hbase.cluster.distributed": "true",
+ "hbase.master.info.port": "0",
+ "hbase.master.port": "0",
+ "hbase.master.startup.retainassign": "true",
+ "hbase.regionserver.hlog.tolerable.lowreplication": "1",
+ "hbase.regionserver.info.port": "0",
+ "hbase.regionserver.port": "0",
+ "hbase.rootdir": "hdfs://sandbox:8020/user/stevel/.slider/cluster/test_cluster_lifecycle/database",
+ "hbase.tmp.dir": "./hbase-tmp",
+ "hbase.zookeeper.property.clientPort": "2181",
+ "hbase.zookeeper.quorum": "sandbox",
+ "slider.template.origin": "hdfs://sandbox:8020/user/stevel/.slider/cluster/test_cluster_lifecycle/snapshot/hbase-site.xml",
+ "slider.unused.option": "1",
+ "zookeeper.znode.parent": "/yarnapps_slider_stevel_test_cluster_lifecycle"
+ }
+}
Added: incubator/slider/site/content/docs/configuration/proposed-hbase.json
URL: http://svn.apache.org/viewvc/incubator/slider/site/content/docs/configuration/proposed-hbase.json?rev=1608561&view=auto
==============================================================================
--- incubator/slider/site/content/docs/configuration/proposed-hbase.json (added)
+++ incubator/slider/site/content/docs/configuration/proposed-hbase.json Mon Jul 7 19:03:38 2014
@@ -0,0 +1,273 @@
+{
+ "version": "2.0.0",
+ "name": "test_cluster_lifecycle",
+ "valid`": true,
+
+ "slider-internal":{
+ "type": "hbase",
+ "createTime": 1393512091276,
+ "updateTime": 1393512117286,
+ "originConfigurationPath": "hdfs://sandbox:8020/user/slider/.slider/cluster/test_cluster_lifecycle/snapshot",
+ "generatedConfigurationPath": "hdfs://sandbox:8020/user/slider/.slider/cluster/test_cluster_lifecycle/generated",
+ "dataPath": "hdfs://sandbox:8020/user/slider/.slider/cluster/test_cluster_lifecycle/database",
+ "slider.tmp.dir": "hdfs://sandbox:8020/user/slider/.slider/cluster/test_cluster_lifecycle/tmp/am",
+ "slider.cluster.directory.permissions": "0770",
+ "slider.data.directory.permissions": "0770"
+ },
+
+ "options": {
+ "slider.am.monitoring.enabled": "false",
+ "slider.cluster.application.image.path": "hdfs://sandbox:8020/hbase.tar.gz",
+ "slider.container.failure.threshold": "5",
+ "slider.container.failure.shortlife": "60",
+ "zookeeper.port": "2181",
+ "zookeeper.path": "/yarnapps_slider_slider_test_cluster_lifecycle",
+ "zookeeper.hosts": "sandbox",
+ "site.hbase.master.startup.retainassign": "true",
+ "site.fs.defaultFS": "hdfs://sandbox:8020",
+ "site.fs.default.name": "hdfs://sandbox:8020",
+ "env.MALLOC_ARENA_MAX": "4",
+ "site.hbase.master.info.port": "0",
+ "site.hbase.regionserver.info.port": "0"
+ },
+
+ "diagnostics": {
+ "create.hadoop.deployed.info": "(detached from release-2.3.0) @dfe46336fbc6a044bc124392ec06b85",
+ "create.application.build.info": "Slider Core-0.13.0-SNAPSHOT Built against commit# 1a94ee4aa1 on Java 1.7.0_45 by slider",
+ "create.hadoop.build.info": "2.3.0",
+ "create.time.millis": "1393512091276",
+ "create.time": "27 Feb 2014 14:41:31 GMT"
+ },
+
+ "info": {
+ "slider.am.restart.supported": "false",
+ "live.time": "27 Feb 2014 14:41:56 GMT",
+ "live.time.millis": "1393512116881",
+ "status.time": "27 Feb 2014 14:42:08 GMT",
+ "status.time.millis": "1393512128726",
+ "yarn.vcores": "32",
+ "yarn.memory": "2048",
+ "status.application.build.info": "Slider Core-0.13.0-SNAPSHOT Built against commit# 1a94ee4aa1 on Java 1.7.0_45 by slider",
+ "status.hadoop.build.info": "2.3.0",
+ "status.hadoop.deployed.info": "bigwheel-m16-2.2.0 @704f1e463ebc4fb89353011407e965"
+ },
+
+ "statistics": {
+
+ "cluster": {
+ "containers.unknown.completed": 0,
+ "containers.start.completed": 3,
+ "containers.live": 1,
+ "containers.start.failed": 0,
+ "containers.failed": 0,
+ "containers.completed": 0,
+ "containers.surplus": 0
+
+ },
+ "roles": {
+ "worker": {
+ "containers.start.completed": 0,
+ "containers.live": 2,
+ "containers.start.failed": 0,
+ "containers.active.requests": 0,
+ "containers.failed": 0,
+ "containers.completed": 0,
+ "containers.desired": 2,
+ "containers.requested": 0
+ },
+ "master": {
+ "containers.start.completed": 0,
+ "containers.live": 1,
+ "containers.start.failed": 0,
+ "containers.active.requests": 0,
+ "containers.failed": 0,
+ "containers.completed": 0,
+ "containers.desired": 1,
+ "containers.requested": 0
+ }
+ }
+ },
+
+ "instances": {
+ "slider": [ "container_1393511571284_0002_01_000001" ],
+ "master": [ "container_1393511571284_0002_01_000003" ],
+ "worker": [
+ "container_1393511571284_0002_01_000002",
+ "container_1393511571284_0002_01_000004"
+ ]
+ },
+
+ "roles": {
+ "worker": {
+ "yarn.memory": "768",
+ "role.instances": "0",
+ "role.name": "worker",
+ "jvm.heapsize": "512M",
+ "yarn.vcores": "1"
+ },
+ "slider": {
+ "yarn.memory": "256",
+ "role.instances": "1",
+ "role.name": "slider",
+ "jvm.heapsize": "256M",
+ "yarn.vcores": "1"
+ },
+ "master": {
+ "yarn.memory": "1024",
+ "role.instances": "0",
+ "role.name": "master",
+ "jvm.heapsize": "512M",
+ "yarn.vcores": "1"
+ }
+ },
+
+
+ "clientProperties": {
+ "fs.defaultFS": "hdfs://sandbox:8020",
+ "hbase.cluster.distributed": "true",
+ "hbase.master.info.port": "0",
+ "hbase.master.port": "0",
+ "hbase.master.startup.retainassign": "true",
+ "hbase.regionserver.hlog.tolerable.lowreplication": "1",
+ "hbase.regionserver.info.port": "0",
+ "hbase.regionserver.port": "0",
+ "hbase.rootdir": "hdfs://sandbox:8020/user/slider/.slider/cluster/test_cluster_lifecycle/database",
+ "hbase.tmp.dir": "./hbase-tmp",
+ "hbase.zookeeper.property.clientPort": "2181",
+ "hbase.zookeeper.quorum": "sandbox",
+ "zookeeper.znode.parent": "/yarnapps_slider_slider_test_cluster_lifecycle"
+ },
+
+
+ "clientfiles": {
+ "hbase-site.xml": "site information for HBase",
+ "log4.properties": "log4.property file"
+ },
+
+ "provider":{
+ "load":0.4,
+ "urls": {
+ "master": ["http://node4:28209"],
+ "worker": ["http://node4:28717", "http://node6:31268"]
+ }
+ },
+
+ "status": {
+ "live": {
+ "worker": {
+ "container_1394032374441_0001_01_000003": {
+ "name": "container_1394032374441_0001_01_000003",
+ "role": "worker",
+ "roleId": 1,
+ "createTime": 1394032384451,
+ "startTime": 1394032384503,
+ "released": false,
+ "host": "192.168.1.88",
+ "state": 3,
+ "exitCode": 0,
+ "command": "hbase-0.98.0/bin/hbase --config $PROPAGATED_CONFDIR regionserver start 1><LOG_DIR>/region-server.txt 2>&1 ; ",
+ "diagnostics": "",
+ "environment": [
+ "HADOOP_USER_NAME=\"slider\"",
+ "HBASE_LOG_DIR=\"/tmp/slider-slider\"",
+ "HBASE_HEAPSIZE=\"256\"",
+ "MALLOC_ARENA_MAX=\"4\"",
+ "PROPAGATED_CONFDIR=\"$PWD/propagatedconf\""
+ ]
+ },
+ "container_1394032374441_0001_01_000002": {
+ "name": "container_1394032374441_0001_01_000002",
+ "role": "worker",
+ "roleId": 1,
+ "createTime": 1394032384451,
+ "startTime": 1394032384552,
+ "released": false,
+ "host": "192.168.1.86",
+ "state": 3,
+ "exitCode": 0,
+ "command": "hbase-0.98.0/bin/hbase --config $PROPAGATED_CONFDIR regionserver start 1><LOG_DIR>/region-server.txt 2>&1 ; ",
+ "diagnostics": "",
+ "environment": [
+ "HADOOP_USER_NAME=\"slider\"",
+ "HBASE_LOG_DIR=\"/tmp/slider-slider\"",
+ "HBASE_HEAPSIZE=\"256\"",
+ "MALLOC_ARENA_MAX=\"4\"",
+ "PROPAGATED_CONFDIR=\"$PWD/propagatedconf\""
+ ]
+ }
+ },
+ "slider": {
+ "container_1394032374441_0001_01_000001": {
+ "name": "container_1394032374441_0001_01_000001",
+ "role": "slider",
+ "roleId": 0,
+ "createTime": 0,
+ "startTime": 0,
+ "released": false,
+ "host": "slider-8.local",
+ "state": 3,
+ "exitCode": 0,
+ "command": "",
+ "diagnostics": ""
+ }
+ },
+ "master": {
+ "container_1394032374441_0001_01_000004": {
+ "name": "container_1394032374441_0001_01_000004",
+ "role": "master",
+ "roleId": 2,
+ "createTime": 1394032384451,
+ "startTime": 1394032384573,
+ "released": false,
+ "host": "192.168.1.86",
+ "state": 3,
+ "exitCode": 0,
+ "command": "hbase-0.98.0/bin/hbase --config $PROPAGATED_CONFDIR master start 1><LOG_DIR>/master.txt 2>&1 ; ",
+ "diagnostics": "",
+ "environment": [
+ "HADOOP_USER_NAME=\"slider\"",
+ "HBASE_LOG_DIR=\"/tmp/slider-slider\"",
+ "HBASE_HEAPSIZE=\"256\"",
+ "MALLOC_ARENA_MAX=\"4\"",
+ "PROPAGATED_CONFDIR=\"$PWD/propagatedconf\""
+ ]
+ }
+ }
+ },
+ "failed": {
+
+ },
+
+ "rolestatus": {
+ "worker": {
+ "role.instances": "2",
+ "role.requested.instances": "0",
+ "role.failed.starting.instances": "0",
+ "role.actual.instances": "2",
+ "role.releasing.instances": "0",
+ "role.failed.instances": "1"
+ },
+ "slider": {
+ "role.instances": "1",
+ "role.requested.instances": "0",
+ "role.name": "slider",
+ "role.actual.instances": "1",
+ "role.releasing.instances": "0",
+ "role.failed.instances": "0"
+ },
+ "master": {
+ "role.instances": "1",
+ "role.requested.instances": "1",
+ "role.name": "master",
+ "role.failed.starting.instances": "0",
+ "role.actual.instances": "0",
+ "role.releasing.instances": "0",
+ "role.failed.instances": "0"
+ }
+ }
+ }
+
+
+
+
+}
Added: incubator/slider/site/content/docs/configuration/redesign.html
URL: http://svn.apache.org/viewvc/incubator/slider/site/content/docs/configuration/redesign.html?rev=1608561&view=auto
==============================================================================
--- incubator/slider/site/content/docs/configuration/redesign.html (added)
+++ incubator/slider/site/content/docs/configuration/redesign.html Mon Jul 7 19:03:38 2014
@@ -0,0 +1,603 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE- 2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <link href="/css/bootstrap.min.css" rel="stylesheet">
+ <link href="/css/bootstrap-theme.min.css" rel="stylesheet">
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
+ <script src="https://oss.maxcdn.com/libs/respond.js/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <link href="/css/slider.css" rel="stylesheet" type="text/css">
+ <link href="//netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
+ <title></title>
+
+</head>
+
+<body>
+<nav class="navbar navbar-default navbar-fixed-top" role="navigation">
+<div class="container-fluid">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#navbar-items">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a class="navbar-brand" href="/index.html">Slider</a>
+ </div>
+ <div class="collapse navbar-collapse" id="navbar-items">
+ <ul class="nav navbar-nav">
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">
+ Project <span class="caret"></span>
+ </a>
+<ul class="dropdown-menu">
+<li id="nav_index"><a href="/index.html">Home</a></li>
+<li id="nav_downloads"><a href="/downloads">Downloads</a></li>
+<li id="nav_source"><a href="https://git-wip-us.apache.org/repos/asf/incubator-slider.git"><i class="fa fa-external-link"></i> Source</a></li>
+<li><a href="http://www.apache.org/licenses/LICENSE-2.0"><i class="fa fa-external-link"></i> License</a></li>
+<li class="divider"></li>
+<li class="dropdown-header">Community</li>
+<li id="nav_team"><a href="/team.html">Team</a></li>
+<li id="nav_mailinglists"><a href="/mailing_lists.html">Mailing Lists</a></li>
+<li class="divider"></li>
+<li class="dropdown-header">Tools</li>
+<li id="nav_jira"><a href="https://issues.apache.org/jira/browse/SLIDER"><i class="fa fa-external-link"></i> Issues</a></li>
+<li id="nav_builds"><a href="https://builds.apache.org/job/Slider-develop"><i class="fa fa-external-link"></i> Builds</a></li>
+</ul>
+</li>
+
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="/design">
+ Design <span class="caret"></span>
+ </a>
+<ul class="dropdown-menu">
+<li id="nav_design_overview"><a href="/design/architecture.html">Overview</a></li>
+<li id="nav_app_needs"><a href="/docs/slider_specs/application_needs.html">Application Needs</a></li>
+<li id="nav_spec"><a href="/design/specification/index.html">Expected Behavior</a></li>
+<li id="nav_registry"><a href="/design/registry/index.html">Service Registry</a></li>
+<li id="nav_role_history"><a href="/design/rolehistory.html">Role History</a></li>
+</ul>
+</li>
+
+
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="/developing">
+ Developing <span class="caret"></span>
+ </a>
+<ul class="dropdown-menu">
+<li id="nav_dev_building"><a href="/developing/building.html">Building</a></li>
+<li id="nav_dev_debugging"><a href="/docs/debugging.html">Debugging</a></li>
+<li id="nav_dev_testing"><a href="/developing/testing.html">Testing</a></li>
+<li id="nav_dev_funtesting"><a href="/developing/functional_tests.html">Functional Testing</a></li>
+<li id="nav_dev_manual_testing"><a href="/developing/manual_testing.html">Manual Testing</a></li>
+<li id="nav_releasing"><a href="/developing/releasing.html">Releasing</a></li>
+</ul>
+</li>
+
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">
+ Using <span class="caret"></span>
+ </a>
+<ul class="dropdown-menu">
+<li id="nav_use_getting_started"><a href="/docs/getting_started.html">Getting Started</a></li>
+<li id="nav_use_manpage"><a href="/docs/manpage.html">Man Page</a></li>
+<li id="nav_use_examples"><a href="/docs/examples.html">Examples</a></li>
+<li id="nav_use_client_config"><a href="/docs/client-configuration.html">Client Configuration</a></li>
+<li id="nav_use_exitcodes"><a href="/docs/exitcodes.html">Client Exit Codes</a></li>
+<li id="nav_use_security"><a href="/docs/security.html">Security</a></li>
+<li id="nav_use_app_package"><a href="/docs/slider_specs/index.html">App Packaging</a></li>
+<li id="nav_use_app_configuration"><a href="/docs/configuration/index.html">App Configuration Model</a></li>
+<li id="nav_use_debugging"><a href="/docs/debugging.html">Debugging</a></li>
+<li id="nav_use_troubleshooting"><a href="/docs/troubleshooting.html">Troubleshooting</a></li>
+</ul>
+</li>
+
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">
+ ASF Links <span class="caret"></span>
+ </a>
+<ul class="dropdown-menu">
+<li><a href="http://www.apache.org"><i class="fa fa-external-link"></i> Apache Software Foundation</a></li>
+<li><a href="http://www.apache.org/foundation/sponsorship.html"><i class="fa fa-external-link"></i> Sponsorship</a></li>
+<li><a href="http://www.apache.org/security/"><i class="fa fa-external-link"></i> Security</a></li>
+<li><a href="http://www.apache.org/foundation/thanks.html"><i class="fa fa-external-link"></i> Thanks</a></li>
+</ul>
+</li>
+</ul>
+
+ </div>
+
+</div>
+</nav>
+
+
+<div class="container-fluid">
+<div class="row">
+ <div class="col-md-2" id="sidebar">
+ <div style="text-align: center">
+ <h1><a href="/index.html">Apache Slider (incubating)</a></h1>
+ <hr>
+Latest release: <strong>0.30-incubating</strong><br>
+ <br>
+ <a id="download-button-sidebar" class="btn btn-success btn-block" href="/downloads/" role="button">Download</a>
+ </div>
+ <hr>
+ <a href="http://incubator.apache.org"><img id="logo" alt="Apache Incubator ™" class="img-responsive" src="http://incubator.apache.org/images/apache-incubator-logo.png"/></a>
+ </div>
+ <div class="col-md-8 col-md-offset-1">
+
+ <div id="content">
+
+ <h1 class="title"></h1>
+
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<h1 id="apache-slider-cluster-specification">Apache Slider Cluster Specification</h1>
+<h3 id="notation">Notation:</h3>
+<p>In this document, a full path to a value is represented as a path
+<code>options/zookeeper.port</code> ; an assigment as <code>options/zookeeper.port=2181</code>.</p>
+<p>A wildcard indicates all entries matching a path: <code>options/zookeeper.*</code>
+or <code>/roles/*/yarn.memory</code></p>
+<h2 id="history">History</h2>
+<p>The Slider cluster specification was implicitly defined in the file
+<code>org.apache.slider.api.ClusterDescription</code>. It had a number of roles</p>
+<ol>
+<li>Persistent representaton of cluster state</li>
+<li>Internal model of desired cluster state within the Application Master.</li>
+<li>Dynamic representation of current cluster state when the AM
+was queried, marshalled over the network as JSON.</li>
+<li>Description of updated state when reconfiguring a running cluster.</li>
+</ol>
+<p>Initially the dynamic status included a complete history of all containers
+-this soon highlit some restrictions on the maximum size of a JSON-formatted
+string in Hadoop's "classic" RPC: 32K, after which the string was silently
+truncated. Accordingly, this history was dropped.</p>
+<p>Having moved to Protocol Buffers as the IPC wire format, with a web view
+alongside, this history could be reconsidered.</p>
+<p>The initial design place most values into the root entry, and relied
+on Jaxon introspection to set and retrieve the values -it was a
+Java-first specification, with no external specificatin or regression tests.</p>
+<p>As the number of entries in the root increased, the design switched to storing
+more attributes into specific sections <em>under</em> the root path:</p>
+<ul>
+<li><code>info</code>: read-only information about the cluster.</li>
+<li><code>statistics</code>: Numeric statistics about the cluster</li>
+</ul>
+<h1 id="sections">Sections</h1>
+<h2 id="root">Root</h2>
+<p>Contains various string and integer values</p>
+<pre class="codehilite"><code>"version": "1.0",
+"name": "test_cluster_lifecycle",
+"type": "hbase",
+"state": 3,
+"createTime": 1393512091276,
+"updateTime": 1393512117286,
+"originConfigurationPath": "hdfs://sandbox:8020/user/stevel/.slider/cluster/test_cluster_lifecycle/snapshot",
+"generatedConfigurationPath": "hdfs://sandbox:8020/user/stevel/.slider/cluster/test_cluster_lifecycle/generated",
+"dataPath": "hdfs://sandbox:8020/user/stevel/.slider/cluster/test_cluster_lifecycle/database",</code></pre>
+
+
+<ul>
+<li>
+<p><code>version</code>: version of the JSON file. Not currently used
+to validate version compatibility; at this point in time
+releases may not be able to read existing .json files.</p>
+</li>
+<li>
+<p><code>name</code>: cluster name</p>
+</li>
+<li><code>type</code>: reference to the provider type -this triggers a Hadoop configuration
+property lookup to find the implementation classes.</li>
+<li><code>state</code>: an enumeration value of the cluster state.<pre class="codehilite"><code>int STATE_INCOMPLETE = 0;
+int STATE_SUBMITTED = 1;
+int STATE_CREATED = 2;
+int STATE_LIVE = 3;
+int STATE_STOPPED = 4;
+int STATE_DESTROYED = 5;</code></pre>
+
+
+</li>
+</ul>
+<p>Only two states are persisted, "incomplete" and "created", though more
+ are used internally.
+ The <code>incomplete</code> state is used during cluster create/build,
+ allowing an incomplete JSON file to be written
+ -so minimising the window for race conditions on cluster construction.</p>
+<ul>
+<li><code>createTime</code> and <code>updateTime</code>: timestamps, informative only.
+ The <code>createTime</code> value is duplicated in <code>/info/createTimeMillis</code></li>
+<li><code>originConfigurationPath</code>, <code>generatedConfigurationPath</code>, <code>dataPath</code> paths
+used internally -if changed the cluster may not start.</li>
+</ul>
+<p><em>Proposed</em>:
+1. Move all state bar <code>name</code> and cluster state
+into a section <code>/slider-internal</code>.
+1. The cluster state is moved from an enum to a simple
+ boolean, <code>valid</code>, set to true when the cluster JSON
+ has been fully constructed.</p>
+<h2 id="info"><code>/info</code></h2>
+<p>Read-only list of information about the application. Generally this is
+intended to be used for debugging and testing.</p>
+<h3 id="persisted-values-static-information-about-the-file-history">Persisted values: static information about the file history</h3>
+<pre class="codehilite"><code>"info" : {
+ "create.hadoop.deployed.info" : "(detached from release-2.3.0) @dfe46336fbc6a044bc124392ec06b85",
+ "create.application.build.info" : "Slider Core-0.13.0-SNAPSHOT Built against commit# 1a94ee4aa1 on Java 1.7.0_45 by stevel",
+ "create.hadoop.build.info" : "2.3.0",
+ "create.time.millis" : "1393512091276",
+},</code></pre>
+
+
+<p><em>Proposed</em>: move persisted info K-V pairs to a section <code>/diagnostics</code>.</p>
+<h3 id="dynamic-values">Dynamic values:</h3>
+<p>whether the AM supports service restart without killing all the containers hosting
+ the role instances:</p>
+<pre class="codehilite"><code>"slider.am.restart.supported" : "false",</code></pre>
+
+
+<p>timestamps of the cluster going live, and when the status query was made</p>
+<pre class="codehilite"><code>"live.time" : "27 Feb 2014 14:41:56 GMT",
+"live.time.millis" : "1393512116881",
+"status.time" : "27 Feb 2014 14:42:08 GMT",
+"status.time.millis" : "1393512128726",</code></pre>
+
+
+<p>yarn data provided to the AM</p>
+<pre class="codehilite"><code>"yarn.vcores" : "32",
+"yarn.memory" : "2048",</code></pre>
+
+
+<p>information about the application and hadoop versions in use. Here
+ the application was built using Hadoop 2.3.0, but is running against the version
+ of Hadoop built for HDP-2.</p>
+<pre class="codehilite"><code>"status.application.build.info" : "Slider Core-0.13.0-SNAPSHOT Built against commit# 1a94ee4aa1 on Java 1.7.0_45 by stevel",
+"status.hadoop.build.info" : "2.3.0",
+"status.hadoop.deployed.info" : "bigwheel-m16-2.2.0 @704f1e463ebc4fb89353011407e965"</code></pre>
+
+
+<p>## <code>instances</code></p>
+<p>Information about the live containers in a cluster</p>
+<pre class="codehilite"><code> "instances": {
+ "slider": [ "container_1393511571284_0002_01_000001" ],
+ "master": [ "container_1393511571284_0002_01_000003" ],
+ "worker": [
+ "container_1393511571284_0002_01_000002",
+ "container_1393511571284_0002_01_000004"
+ ]
+ },</code></pre>
+
+
+<p>There's no information about location, nor is there any history about containers
+that are no longer part of the cluster (i.e. failed & released containers). </p>
+<p>It could be possible to include a list of previous containers,
+though Slider would need to be selective about how many to store
+(or how much detail to retain) on those previous containers.</p>
+<p>Perhaps the list could be allowed to grow without limit, but detail
+only preserved on the last 100. If more containers fail than that,
+there is likely to be a problem which the most recent containers
+will also display.</p>
+<p><em>Proposed</em> </p>
+<ol>
+<li>Return to the full serialization of container state -but only for running containers.</li>
+<li>Have a list of failed containers, but only include last 8; make it a rolling
+buffer. This avoids a significantly failing role to overload the status document.</li>
+</ol>
+<p>## <code>statistics</code></p>
+<p>Statistics on each role. </p>
+<p>They can be divided into counters that only increase</p>
+<pre class="codehilite"><code>"containers.start.completed": 0,
+"containers.start.failed": 0,
+"containers.failed": 0,
+"containers.completed": 0,
+"containers.requested": 0</code></pre>
+
+
+<p>and those that vary depending upon the current state</p>
+<pre class="codehilite"><code>"containers.live": 0,
+"containers.active.requests": 0,
+"containers.desired": 0,</code></pre>
+
+
+<ul>
+<li>
+<p>Propose: move these values out of statistics into some other section, as they
+are state, not statistics*</p>
+<p>"statistics": {
+ "worker": {
+ "containers.start.completed": 0,
+ "containers.live": 2,
+ "containers.start.failed": 0,
+ "containers.active.requests": 0,
+ "containers.failed": 0,
+ "containers.completed": 0,
+ "containers.desired": 2,
+ "containers.requested": 0
+ },
+ "slider": {
+ "containers.unknown.completed": 0,
+ "containers.start.completed": 3,
+ "containers.live": 1,
+ "containers.start.failed": 0,
+ "containers.failed": 0,
+ "containers.completed": 0,
+ "containers.surplus": 0
+ },
+ "master": {
+ "containers.start.completed": 0,
+ "containers.live": 1,
+ "containers.start.failed": 0,
+ "containers.active.requests": 0,
+ "containers.failed": 0,
+ "containers.completed": 0,
+ "containers.desired": 1,
+ "containers.requested": 0
+ }
+ },</p>
+</li>
+</ul>
+<p>The <code>/statistics/slider</code> section is unusual in that it provides the aggregate statistics
+of the cluster -this is not obvious. A different name could be used -but
+again, there's a risk of clash with or confusion with a role. </p>
+<p>Better to have a specific <code>/statistics/cluster</code> element,
+and to move the roles' statistics under <code>/statistics/roles</code>:</p>
+<pre class="codehilite"><code>"statistics": {
+ "cluster": {
+ "containers.unknown.completed": 0,
+ "containers.start.completed": 3,
+ "containers.live": 1,
+ "containers.start.failed": 0,
+ "containers.failed": 0,
+ "containers.completed": 0,
+ "containers.surplus": 0
+
+ },
+ "roles": {
+ "worker": {
+ "containers.start.completed": 0,
+ "containers.live": 2,
+ "containers.start.failed": 0,
+ "containers.active.requests": 0,
+ "containers.failed": 0,
+ "containers.completed": 0,
+ "containers.desired": 2,
+ "containers.requested": 0
+ },
+ "master": {
+ "containers.start.completed": 0,
+ "containers.live": 1,
+ "containers.start.failed": 0,
+ "containers.active.requests": 0,
+ "containers.failed": 0,
+ "containers.completed": 0,
+ "containers.desired": 1,
+ "containers.requested": 0
+ }
+ }
+},</code></pre>
+
+
+<p>This approach allows extra statistics sections to be added (perhaps
+by providers), without any changes to the toplevel section.</p>
+<h2 id="options">Options</h2>
+<p>A list of options used by Slider and its providers to build up the AM
+and the configurations of the deployed service components</p>
+<pre class="codehilite"><code>"options": {
+ "zookeeper.port": "2181",
+ "site.hbase.master.startup.retainassign": "true",
+ "slider.cluster.application.image.path": "hdfs://sandbox:8020/hbase.tar.gz",
+ "site.fs.defaultFS": "hdfs://sandbox:8020",
+ "slider.container.failure.threshold": "5",
+ "site.fs.default.name": "hdfs://sandbox:8020",
+ "slider.cluster.directory.permissions": "0770",
+ "slider.am.monitoring.enabled": "false",
+ "zookeeper.path": "/yarnapps_slider_stevel_test_cluster_lifecycle",
+ "slider.tmp.dir": "hdfs://sandbox:8020/user/stevel/.slider/cluster/test_cluster_lifecycle/tmp/am",
+ "slider.data.directory.permissions": "0770",
+ "zookeeper.hosts": "sandbox",
+ "slider.container.failure.shortlife": "60"
+},</code></pre>
+
+
+<p>Some for these options have been created by slider itself ("slider.tmp.dir")
+for internal use -and are cluster specific. If/when the ability to use
+an existing json file as a template for a new cluster is added, having these
+options in the configuration will create problems</p>
+<h1 id="proposed-changes">Proposed Changes</h1>
+<h2 id="move-slider-internal-state-to-slider-internal">Move Slider internal state to <code>/slider-internal</code></h2>
+<p>Move all slider "private" data to an internal section,<code>/slider-internal</code>
+including those in the toplevel directory and in <code>/options</code></p>
+<h2 id="allow-options-and-roles-options-entries-to-take-the-value-null">Allow <code>/options</code> and <code>roles/*/</code> options entries to take the value "null".</h2>
+<p>This would be a definition that the value must be defined before the cluster
+can start. Provider templates could declare this.</p>
+<h2 id="make-client-configuration-retrieval-hierarchical-and-maybe-move-out-of-the">Make client configuration retrieval hierarchical -and maybe move out of the</h2>
+<p>status</p>
+<p>The current design assumes that it is a -site.xml file being served up. This
+does not work for alternate file formats generated by the Provider.</p>
+<h2 id="role-options">Role Options</h2>
+<p>The <code>/roles/$ROLENAME/</code> clauses each provide options for a
+specific role.</p>
+<p>This includes
+1. <code>role.instances</code>: defines the number of instances of a role to create
+1. <code>env.</code> environment variables for launching the container
+1. <code>yarn.</code> properties to configure YARN requests.
+1. <code>jvm.heapsize</code>: an option supported by some providers to
+fix the heap size of a component.
+1. <code>app.infoport</code>: an option supported by some providers (e.g. HBase)
+to fix the port to which a role (master or worker) binds its web UI.</p>
+<pre class="codehilite"><code> "worker": {
+ "yarn.memory": "768",
+ "env.MALLOC_ARENA_MAX": "4",
+ "role.instances": "0",
+ "role.name": "worker",
+ "jvm.heapsize": "512M",
+ "yarn.vcores": "1",
+ "app.infoport": "0"
+ },</code></pre>
+
+
+<p>In a live cluster, the role information also includes status information
+about the cluster.</p>
+<pre class="codehilite"><code> "master": {
+ "yarn.memory": "1024",
+ "env.MALLOC_ARENA_MAX": "4",
+ "role.instances": "0",
+ "role.requested.instances": "0",
+ "role.name": "master",
+ "role.failed.starting.instances": "0",
+ "role.actual.instances": "0",
+ "jvm.heapsize": "512M",
+ "yarn.vcores": "1",
+ "role.releasing.instances": "0",
+ "role.failed.instances": "0",
+ "app.infoport": "0"
+ }</code></pre>
+
+
+<p>The role <code>slider</code> represents the Slider Application Master itself.</p>
+<pre class="codehilite"><code> "slider": {
+ "yarn.memory": "256",
+ "env.MALLOC_ARENA_MAX": "4",
+ "role.instances": "1",
+ "role.name": "slider",
+ "jvm.heapsize": "256M",
+ "yarn.vcores": "1",
+ },</code></pre>
+
+
+<h3 id="proposed">Proposed:</h3>
+<ol>
+<li>move all dynamic role status to its own clauses.</li>
+<li>use a simple inheritance model from <code>/options</code></li>
+<li>don't allow role entries to alter the cluster state. </li>
+</ol>
+<h3 id="proposed-clientproperties-continues-return-key-val-pairs">Proposed: <code>/clientProperties</code> continues return Key-val pairs</h3>
+<p>The <code>/clientProperties</code> section will remain, with key-val pairs of type
+string, the expectation being this is where providers can insert specific
+single attributes for client applications.</p>
+<p>These values can be converted to application-specific files on the client,
+in code -as done today in the Slider CLI-, or via template expansion (beyond
+the scope of this document.</p>
+<h3 id="proposed-alongside-clientproperties-comes-clientfiles">Proposed: alongside <code>/clientProperties</code> comes <code>/clientfiles</code></h3>
+<p>This section will list all files that an application instance can generate
+for clients, along with with a description.</p>
+<pre class="codehilite"><code>"/clientfiles/hbase-site.xml": "site information for HBase"
+"/clientfiles/log4.properties": "log4.property file"</code></pre>
+
+
+<p>A new CLI command would be added to retrieve a client file.
+1. The specific file must be named.
+1. If it is not present, an error must be raised.
+1. If it is present, it is downloaded and output to the console/to a named
+destination file/directory <code>--outfile <file></code> and <code>--outdir <dir></code>
+1. If the <code>--list</code> argument is provided, the list of available files is
+returned (e.g.) </p>
+<pre class="codehilite"><code>hbase-site.xml: site information for HBase
+log4.properties: log4.property file</code></pre>
+
+
+<p><em>No attempt to parse/process the body of the messages will be returned.</em></p>
+<p>In a REST implementation of the client API, /clientconf would be a path
+to the list of options; each file a path underneath.</p>
+<p>Client configuration file retrieval outside the status completely;
+the status just lists the possible values; a separate call returns them.</p>
+<p>This will permit binary content to be retrieved, and avoid any marshalling
+problems and inefficiencies.</p>
+<p>With this change, there will now be two ways to generate client configuration
+files</p>
+<ul>
+<li>Client-side: as today</li>
+<li>Server-side: via the provider</li>
+</ul>
+<p>Client side is more extensible as it allows for arbitrary clients; server-side
+is restricted to those files which the application provider is capable of
+generating. The advantage of the server-side option is that for those files
+about which the provider is aware of, they will be visible through the
+REST and Web UIs, so trivially retrieved.</p>
+<h3 id="stop-intermixing-role-specification-with-role-current-state">Stop intermixing role specification with role current state</h3>
+<p>Create a new section, <code>rolestatus</code>, which lists the current status
+of the roles: how many are running vs requested, how many are being
+released.</p>
+<p>There's some overlap here with the <code>/statistics</code> field, so we should
+either merge them or clearly separate the two. Only the <code>role.failed</code>
+properties match entries in the statistics -perhaps they should be cut.</p>
+<h4 id="provider-specific-status">provider-specific status</h4>
+<p>Allow providers to publish information to the status, in their
+own section.</p>
+<p>There already is support for providers updating the cluster status
+in Slider 12.1 and earlier, but it has flaws</p>
+<p>A key one is that it is done sychronously on a <code>getStatus()</code> call;
+as providers may perform a live query of their status (example, the HBase
+provider looks up the Web UI ports published by HBase to zookeeper),
+there's overhead, and if the operation blocks (example: when HBase hasn't
+ever been deployed and the zookeeper path is empty), then the status
+call blocks.</p>
+<p><em>Proposed:</em></p>
+<ol>
+<li>There is a specific <code>/provider</code> section</li>
+<li>There's no restriction on what JSON is permitted in this section.</li>
+<li>Providers may make their own updates to the application state to read and
+write this block -operations that are asynchronous to any status queries.</li>
+</ol>
+ </div>
+
+ <div id="footer">
+ <a alt="Apache Software Foundation" href="http://www.apache.org">
+ <img id="asf-logo" alt="Apache Software Foundation" src="/images/feather-small.gif" width="100">
+ </a>
+ <div class="copyright">
+ <p>
+ Copyright © 2011-2014 The Apache Software Foundation, Licensed under
+ the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.
+ Apache Slider, Slider, Apache, the Apache feather logo, and the Apache Incubator
+ project logo are trademarks of the <a href="http://www.apache.org">Apache Software Foundation</a>.<br />
+ Site created with <a href="http://getbootstrap.com/">Bootstrap</a> including icons from <a href="http://glyphicons.com/">GLYPHICONS</a> and <a href="http://fontawesome.io/">Font Awesome</a>.
+ </p>
+ </div>
+ </div>
+ </div>
+
+</div>
+</div>
+
+<script type="text/javascript">
+
+</script>
+</body>
+</html>
Propchange: incubator/slider/site/content/docs/configuration/redesign.html
------------------------------------------------------------------------------
svn:eol-style = native
Added: incubator/slider/site/content/docs/configuration/resolved-resources.json
URL: http://svn.apache.org/viewvc/incubator/slider/site/content/docs/configuration/resolved-resources.json?rev=1608561&view=auto
==============================================================================
--- incubator/slider/site/content/docs/configuration/resolved-resources.json (added)
+++ incubator/slider/site/content/docs/configuration/resolved-resources.json Mon Jul 7 19:03:38 2014
@@ -0,0 +1,22 @@
+{
+ "schema": "http://example.org/specification/v2.0.0",
+
+ "metadata": {
+ "description": "example of a resources file"
+ },
+
+ "global": {
+ "yarn.vcores": "1",
+ "yarn.memory": "512"
+ },
+
+ "components": {
+ "master": {
+ "instances": "1",
+ "yarn.memory": "1024"
+ },
+ "worker": {
+ "instances":"5"
+ }
+ }
+}
\ No newline at end of file
Added: incubator/slider/site/content/docs/configuration/specification.html
URL: http://svn.apache.org/viewvc/incubator/slider/site/content/docs/configuration/specification.html?rev=1608561&view=auto
==============================================================================
--- incubator/slider/site/content/docs/configuration/specification.html (added)
+++ incubator/slider/site/content/docs/configuration/specification.html Mon Jul 7 19:03:38 2014
@@ -0,0 +1,629 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE- 2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <link href="/css/bootstrap.min.css" rel="stylesheet">
+ <link href="/css/bootstrap-theme.min.css" rel="stylesheet">
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
+ <script src="https://oss.maxcdn.com/libs/respond.js/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <link href="/css/slider.css" rel="stylesheet" type="text/css">
+ <link href="//netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
+ <title></title>
+
+</head>
+
+<body>
+<nav class="navbar navbar-default navbar-fixed-top" role="navigation">
+<div class="container-fluid">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#navbar-items">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a class="navbar-brand" href="/index.html">Slider</a>
+ </div>
+ <div class="collapse navbar-collapse" id="navbar-items">
+ <ul class="nav navbar-nav">
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">
+ Project <span class="caret"></span>
+ </a>
+<ul class="dropdown-menu">
+<li id="nav_index"><a href="/index.html">Home</a></li>
+<li id="nav_downloads"><a href="/downloads">Downloads</a></li>
+<li id="nav_source"><a href="https://git-wip-us.apache.org/repos/asf/incubator-slider.git"><i class="fa fa-external-link"></i> Source</a></li>
+<li><a href="http://www.apache.org/licenses/LICENSE-2.0"><i class="fa fa-external-link"></i> License</a></li>
+<li class="divider"></li>
+<li class="dropdown-header">Community</li>
+<li id="nav_team"><a href="/team.html">Team</a></li>
+<li id="nav_mailinglists"><a href="/mailing_lists.html">Mailing Lists</a></li>
+<li class="divider"></li>
+<li class="dropdown-header">Tools</li>
+<li id="nav_jira"><a href="https://issues.apache.org/jira/browse/SLIDER"><i class="fa fa-external-link"></i> Issues</a></li>
+<li id="nav_builds"><a href="https://builds.apache.org/job/Slider-develop"><i class="fa fa-external-link"></i> Builds</a></li>
+</ul>
+</li>
+
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="/design">
+ Design <span class="caret"></span>
+ </a>
+<ul class="dropdown-menu">
+<li id="nav_design_overview"><a href="/design/architecture.html">Overview</a></li>
+<li id="nav_app_needs"><a href="/docs/slider_specs/application_needs.html">Application Needs</a></li>
+<li id="nav_spec"><a href="/design/specification/index.html">Expected Behavior</a></li>
+<li id="nav_registry"><a href="/design/registry/index.html">Service Registry</a></li>
+<li id="nav_role_history"><a href="/design/rolehistory.html">Role History</a></li>
+</ul>
+</li>
+
+
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="/developing">
+ Developing <span class="caret"></span>
+ </a>
+<ul class="dropdown-menu">
+<li id="nav_dev_building"><a href="/developing/building.html">Building</a></li>
+<li id="nav_dev_debugging"><a href="/docs/debugging.html">Debugging</a></li>
+<li id="nav_dev_testing"><a href="/developing/testing.html">Testing</a></li>
+<li id="nav_dev_funtesting"><a href="/developing/functional_tests.html">Functional Testing</a></li>
+<li id="nav_dev_manual_testing"><a href="/developing/manual_testing.html">Manual Testing</a></li>
+<li id="nav_releasing"><a href="/developing/releasing.html">Releasing</a></li>
+</ul>
+</li>
+
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">
+ Using <span class="caret"></span>
+ </a>
+<ul class="dropdown-menu">
+<li id="nav_use_getting_started"><a href="/docs/getting_started.html">Getting Started</a></li>
+<li id="nav_use_manpage"><a href="/docs/manpage.html">Man Page</a></li>
+<li id="nav_use_examples"><a href="/docs/examples.html">Examples</a></li>
+<li id="nav_use_client_config"><a href="/docs/client-configuration.html">Client Configuration</a></li>
+<li id="nav_use_exitcodes"><a href="/docs/exitcodes.html">Client Exit Codes</a></li>
+<li id="nav_use_security"><a href="/docs/security.html">Security</a></li>
+<li id="nav_use_app_package"><a href="/docs/slider_specs/index.html">App Packaging</a></li>
+<li id="nav_use_app_configuration"><a href="/docs/configuration/index.html">App Configuration Model</a></li>
+<li id="nav_use_debugging"><a href="/docs/debugging.html">Debugging</a></li>
+<li id="nav_use_troubleshooting"><a href="/docs/troubleshooting.html">Troubleshooting</a></li>
+</ul>
+</li>
+
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">
+ ASF Links <span class="caret"></span>
+ </a>
+<ul class="dropdown-menu">
+<li><a href="http://www.apache.org"><i class="fa fa-external-link"></i> Apache Software Foundation</a></li>
+<li><a href="http://www.apache.org/foundation/sponsorship.html"><i class="fa fa-external-link"></i> Sponsorship</a></li>
+<li><a href="http://www.apache.org/security/"><i class="fa fa-external-link"></i> Security</a></li>
+<li><a href="http://www.apache.org/foundation/thanks.html"><i class="fa fa-external-link"></i> Thanks</a></li>
+</ul>
+</li>
+</ul>
+
+ </div>
+
+</div>
+</nav>
+
+
+<div class="container-fluid">
+<div class="row">
+ <div class="col-md-2" id="sidebar">
+ <div style="text-align: center">
+ <h1><a href="/index.html">Apache Slider (incubating)</a></h1>
+ <hr>
+Latest release: <strong>0.30-incubating</strong><br>
+ <br>
+ <a id="download-button-sidebar" class="btn btn-success btn-block" href="/downloads/" role="button">Download</a>
+ </div>
+ <hr>
+ <a href="http://incubator.apache.org"><img id="logo" alt="Apache Incubator ™" class="img-responsive" src="http://incubator.apache.org/images/apache-incubator-logo.png"/></a>
+ </div>
+ <div class="col-md-8 col-md-offset-1">
+
+ <div id="content">
+
+ <h1 class="title"></h1>
+
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<h1 id="apache-slider-specification-of-the-cluster-description">Apache Slider Specification of the "Cluster Description"</h1>
+<ul>
+<li>This is partially obsolete. Slider still returns the Slider Cluster Description
+as changing it will break most of the unit tests -once these are updated
+this document will be completely obsolete and replaced with a new one.</li>
+</ul>
+<h3 id="notation">Notation:</h3>
+<p>In this document, a full path to a value is represented as a path
+<code>options/zookeeper.port</code> ; an assigment as <code>options/zookeeper.port=2181</code>.</p>
+<p>A wildcard indicates all entries matching a path: <code>options/zookeeper.*</code>
+or <code>/roles/*/yarn.memory</code></p>
+<h2 id="core-concepts">Core Concepts</h2>
+<p>The specificaton of an application instance is defined in an application instance
+directory, <code>${user.home}/.slidera/clusters/${clustername}/cluster.json</code>)</p>
+<h2 id="sections-for-specifying-and-describing-cluster-state">Sections for specifying and describing cluster state</h2>
+<p>The cluster desciption is hierarchal, with standardized sections.</p>
+<p>Different sections have one of three roles.</p>
+<ol>
+<li>
+<p>Storage and specification of internal properties used to define a cluster -properties
+that should not be modified by users -doing so is likely to render the
+cluster undeployable.</p>
+</li>
+<li>
+<p>Storage and specification of the components deployed by Slider.
+These sections define options for the deployed application, the size of
+the deployed application, attributes of the deployed roles, and customizable
+aspects of the Slider application master. </p>
+</li>
+</ol>
+<p>This information defines the <em>desired state</em> of a cluster.</p>
+<p>Users may edit these sections, either via the CLI, or by directly editing the <code>cluster.json</code> file of
+ a frozen cluster.</p>
+<ol>
+<li>Status information provided by a running cluster. These include:
+ information about the cluster, statistics, information about reach role in
+ the cluster -as well as other aspects of the deployment.</li>
+</ol>
+<p>This information describes the <em>actual state</em> of a cluster.</p>
+<p>Using a common format for both the specification and description of a cluster
+may be confusing, but it is designed to unify the logic needed to parse
+and process cluster descriptions. There is only one JSON file to parse
+-merely different sections of relevance at different times.</p>
+<h2 id="role-by-role-subsections">Role-by-role subsections</h2>
+<p>A slider-deployed application consists of the single Slider application master,
+and one or more roles -specific components in the actual application.</p>
+<p>The <code>/roles</code> section contains a listing for each role,
+declaring the number of instances of each role desired,
+possibly along with some details defining the actual execution of the application.</p>
+<p>The <code>/statistics/roles/</code> section returns statistics on each role,
+while <code>/instances</code> has a per-role entry listing the YARN
+containers hosting instances. </p>
+<h2 id="cluster-information-for-applications">Cluster information for applications</h2>
+<p>The AM/application provider may generate information for use by client applications.</p>
+<p>There are three ways to provide this</p>
+<ol>
+<li>A section in which simple key-value pairs are provided for interpretation
+by client applications -usually to generate configuration documents</li>
+<li>A listing of files that may be provided directly to a client. The API to provide these files is not covered by this document.</li>
+<li>A provider-specific section in which arbitrary values and structures may be defined. This allows greater flexibility in the information that a provider can publish -though it does imply custom code to process this data on the client.</li>
+</ol>
+<h1 id="persistent-specification-sections">Persistent Specification Sections</h1>
+<h2 id="root">"/" : root</h2>
+<p>The root contains a limited number of key-value pairs, </p>
+<ul>
+<li>
+<p><code>version</code>: string; required.
+The version of the JSON file, as an <code>x.y.z</code> version string.</p>
+<ol>
+<li>Applications MUST verify that they can process a specific version.</li>
+<li>The version number SHOULD be incremented in the minor "z" value
+after enhancements that are considered backwards compatible.
+Incompatible updates MUST be updated with a new "y" value.
+The final, "x" number, is to be reserved for major reworkings
+of the cluster specification itself (this document or its
+successors).</li>
+</ol>
+</li>
+<li>
+<p><code>name</code>: string; required. Cluster name; </p>
+</li>
+<li><code>type</code>: string; required.
+Reference to the provider type -this triggers a Hadoop configuration
+property lookup to find the implementation classes.</li>
+<li><code>valid</code>: boolean; required.
+Flag to indicate whether or not a specification is considered valid.
+If false, the rest of the document is in an unknown state.</li>
+</ul>
+<h2 id="slider-internal-internal-confiugration"><code>/slider-internal</code>: internal confiugration</h2>
+<p>Stores internal configuration options. These parameters
+are not defined in this document.</p>
+<h2 id="diagnostics-diagnostics-sections"><code>/diagnostics</code>: diagnostics sections</h2>
+<p>Persisted list of information about Slider. </p>
+<p>Static information about the file history</p>
+<pre class="codehilite"><code>"diagnostics" : {
+ "create.hadoop.deployed.info" :
+ "(detached from release-2.3.0) @dfe46336fbc6a044bc124392ec06b85",
+ "create.application.build.info" :
+ "Slider Core-0.13.0-SNAPSHOT Built against commit# 1a94ee4aa1 on Java 1.7.0_45 by stevel",
+ "create.hadoop.build.info" : "2.3.0",
+ "create.time.millis" : "1393512091276",
+},</code></pre>
+
+
+<p>This information is not intended to provide anything other
+than diagnostics to an application; the values and their meaning
+are not defined. All applications MUST be able to process
+an empty or absent <code>/diagnostics</code> section.</p>
+<h2 id="options-cluster-options">Options: cluster options</h2>
+<p>A persisted list of options used by Slider and its providers to build up the AM
+and the configurations of the deployed service components</p>
+<pre class="codehilite"><code>"options": {
+ "slider.am.monitoring.enabled": "false",
+ "slider.cluster.application.image.path": "hdfs://sandbox:8020/hbase.tar.gz",
+ "slider.container.failure.threshold": "5",
+ "slider.container.failure.shortlife": "60",
+ "zookeeper.port": "2181",
+ "zookeeper.path": "/yarnapps_slider_stevel_test_cluster_lifecycle",
+ "zookeeper.hosts": "sandbox",
+ "site.hbase.master.startup.retainassign": "true",
+ "site.fs.defaultFS": "hdfs://sandbox:8020",
+ "site.fs.default.name": "hdfs://sandbox:8020",
+ "env.MALLOC_ARENA_MAX": "4",
+ "site.hbase.master.info.port": "0",
+ "site.hbase.regionserver.info.port": "0"
+},</code></pre>
+
+
+<p>Many of the properties are automatically set by Slider when a cluster is constructed.
+They may be edited afterwards.</p>
+<h3 id="standard-option-types">Standard Option types</h3>
+<p>All option values MUST be strings.</p>
+<h4 id="slider"><code>slider.</code></h4>
+<p>All options that begin with <code>slider.</code> are intended for use by slider and
+providers to configure the Slider application master itself, and the
+application. For example, <code>slider.container.failure.threshold</code> defines
+the number of times a container must fail before the role (and hence the cluster)
+is considered to have failed. As another example, the zookeeper bindings
+such as <code>zookeeper.hosts</code> are read by the HBase and Ambari providers, and
+used to modify the applications' site configurations with application-specific
+properties.</p>
+<h4 id="site"><code>site.</code></h4>
+<p>These are properties that are expected to be propagated to an application's
+ <code>site</code> configuration -if such a configuration is created. For HBase, the
+ site file is <code>hbase-site.xml</code>; for Accumulo it is <code>accumulo-site.xml</code></p>
+<ol>
+<li>The destination property is taken by removing the prefix <code>site.</code>, and
+setting the shortened key with the defined value.</li>
+<li>Not all applications have the notion of a site file; These applications MAY
+ignore the settings.</li>
+<li>Providers MAY validate site settings to recognise invalid values. This
+aids identifying and diagnosing startup problems.</li>
+</ol>
+<h4 id="env"><code>env.</code></h4>
+<p>These are options to configure environment variables in the roles. When
+a container is started, all <code>env.</code> options have the prefix removed, and
+are then set as environment variables in the target context.</p>
+<ol>
+<li>The Slider AM uses these values to configure itself, after following the
+option/role merge process.</li>
+<li>Application providers SHOULD follow the same process.</li>
+</ol>
+<h2 id="roles-role-declarations">'/roles': role declarations</h2>
+<p>The <code>/roles/$ROLENAME/</code> clauses each provide options for a
+specific role.</p>
+<p>This includes
+1. <code>role.instances</code>: defines the number of instances of a role to create
+1. <code>env.</code> environment variables for launching the container
+1. <code>yarn.</code> properties to configure YARN requests.
+1. <code>jvm.heapsize</code>: an option supported by some providers to
+fix the heap size of a component.</p>
+<pre class="codehilite"><code> "worker": {
+ "yarn.memory": "768",
+ "env.MALLOC_ARENA_MAX": "4",
+ "role.instances": "0",
+ "role.name": "worker",
+ "role.failed.starting.instances": "0",
+ "jvm.heapsize": "512M",
+ "yarn.vcores": "1",
+ },</code></pre>
+
+
+<p>The role <code>slider</code> represents the Slider Application Master itself.</p>
+<pre class="codehilite"><code> "slider": {
+ "yarn.memory": "256",
+ "env.MALLOC_ARENA_MAX": "4",
+ "role.instances": "1",
+ "role.name": "slider",
+ "jvm.heapsize": "256M",
+ "yarn.vcores": "1",
+ },</code></pre>
+
+
+<p>Providers may support a fixed number of roles -or they may support a dynamic
+number of roles defined at run-time, potentially from other data sources.</p>
+<h2 id="how-options-and-role-options-are-merged">How <code>/options</code> and role options are merged.</h2>
+<p>The options declared for a specific role are merged with the cluster-wide options
+to define the final options for a role. This is implemented in a simple
+override model: role-specific options can override any site-wide options.</p>
+<ol>
+<li>The options defined in <code>/options</code> are used to create the initial option
+map for each role.</li>
+<li>The role's options are then applied to the map -this may overwrite definitions
+from the <code>/options</code> section.</li>
+<li>There is no way to "undefine" a cluster option, merely overwrite it. </li>
+<li>The merged map is then used by the provider to create the component.</li>
+<li>The special <code>slider</code> role is used in the CLI to define the attributes of the AM.</li>
+</ol>
+<p>Options set on a role do not affect any site-wide options: they
+are specific to the invidual role being created. </p>
+<p>As such, overwriting a <code>site.</code> option may have no effect -or it it may
+change the value of a site configuration document <em>in that specific role instance</em>.</p>
+<h3 id="standard-role-options">Standard role options</h3>
+<ul>
+<li><code>role.instances</code> : number; required.
+ The number of instances of that role desired in the application.</li>
+<li><code>yarn.vcores</code> : number.
+ The number of YARN "virtual cores" to request for each role instance.
+ The larger the number, the more CPU allocation -and potentially the longer
+ time to satisfy the request and so instantiate the node.
+ If the value '"-1"<code>is used -for any role but</code>slider`-the maximum value
+ available to the application is requested.</li>
+<li>
+<p><code>yarn.memory</code> : number.
+ The number in Megabytes of RAM to request for each role instance.
+ The larger the number, the more memory allocation -and potentially the longer
+ time to satisfy the request and so instantiate the node.
+ If the value '"-1"<code>is used -for any role but</code>slider`-the maximum value
+ available to the application is requested.</p>
+</li>
+<li>
+<p><code>env.</code> environment variables.
+String environment variables to use when setting up the container</p>
+</li>
+</ul>
+<h3 id="provider-specific-role-options">Provider-specific role options</h3>
+<ul>
+<li><code>jvm.heapsize</code> -the amount of memory for a provider to allocate for
+ a processes JVM. Example "512M". This option MAY be implemented by a provider.</li>
+</ul>
+<h1 id="dynamic-information-sections">Dynamic Information Sections</h1>
+<p>These are the parts of the document that provide dynamic run-time
+information about an application. They are provided by the
+Slider Application Master when a request for the cluster status is issued.</p>
+<h2 id="info"><code>/info</code></h2>
+<p>Dynamic set of string key-value pairs containing
+information about the running application -as provided by th </p>
+<p>The values in this section are not normatively defined. </p>
+<p>Here are some standard values</p>
+<ul>
+<li>
+<p><code>slider.am.restart.supported"</code> whether the AM supports service restart without killing all the containers hosting
+ the role instances:</p>
+<pre class="codehilite"><code>"slider.am.restart.supported" : "false",</code></pre>
+
+
+</li>
+<li>
+<p>timestamps of the cluster going live, and when the status query was made</p>
+<pre class="codehilite"><code>"live.time" : "27 Feb 2014 14:41:56 GMT",
+"live.time.millis" : "1393512116881",
+"status.time" : "27 Feb 2014 14:42:08 GMT",
+"status.time.millis" : "1393512128726",</code></pre>
+
+
+</li>
+<li>
+<p>yarn data provided to the AM</p>
+<pre class="codehilite"><code>"yarn.vcores" : "32",
+"yarn.memory" : "2048",</code></pre>
+
+
+</li>
+<li>
+<p>information about the application and hadoop versions in use. Here
+ the application was built using Hadoop 2.3.0, but is running against the version
+ of Hadoop built for HDP-2.</p>
+<pre class="codehilite"><code>"status.application.build.info" : "Slider Core-0.13.0-SNAPSHOT Built against commit# 1a94ee4aa1 on Java 1.7.0_45 by stevel",
+"status.hadoop.build.info" : "2.3.0",
+"status.hadoop.deployed.info" : "bigwheel-m16-2.2.0 @704f1e463ebc4fb89353011407e965"</code></pre>
+
+
+</li>
+</ul>
+<p>As with the <code>/diagnostics</code> section, this area is primarily intended
+for debugging.</p>
+<p>## <code>/instances</code>: instance list</p>
+<p>Information about the live containers in a cluster</p>
+<pre class="codehilite"><code> "instances": {
+ "slider": [ "container_1393511571284_0002_01_000001" ],
+ "master": [ "container_1393511571284_0002_01_000003" ],
+ "worker": [
+ "container_1393511571284_0002_01_000002",
+ "container_1393511571284_0002_01_000004"
+ ]
+ },</code></pre>
+
+
+<h2 id="status-detailed-dynamic-state"><code>/status</code>: detailed dynamic state</h2>
+<p>This provides more detail on the application including live and failed instances</p>
+<h3 id="statuslive-live-role-instances-by-container"><code>/status/live</code>: live role instances by container</h3>
+<pre class="codehilite"><code>"cluster": {
+ "live": {
+ "worker": {
+ "container_1394032374441_0001_01_000003": {
+ "name": "container_1394032374441_0001_01_000003",
+ "role": "worker",
+ "roleId": 1,
+ "createTime": 1394032384451,
+ "startTime": 1394032384503,
+ "released": false,
+ "host": "192.168.1.88",
+ "state": 3,
+ "exitCode": 0,
+ "command": "hbase-0.98.0/bin/hbase --config $PROPAGATED_CONFDIR regionserver start 1><LOG_DIR>/region-server.txt 2>&1 ; ",
+ "diagnostics": "",
+ "environment": [
+ "HADOOP_USER_NAME=\"slider\"",
+ "HBASE_LOG_DIR=\"/tmp/slider-slider\"",
+ "HBASE_HEAPSIZE=\"256\"",
+ "MALLOC_ARENA_MAX=\"4\"",
+ "PROPAGATED_CONFDIR=\"$PWD/propagatedconf\""
+ ]
+ }
+ }
+ failed : {}
+ }</code></pre>
+
+
+<p>All live instances MUST be described in <code>/status/live</code></p>
+<p>Failed clusters MAY be listed in the <code>/status/failed</code> section, specifically,
+a limited set of recently failed clusters SHOULD be provided.</p>
+<p>Future versions of this document may introduce more sections under <code>/status</code>.</p>
+<h3 id="statusrolestatus-role-status-information"><code>/status/rolestatus</code>: role status information</h3>
+<p>This lists the current status of the roles:
+How many are running vs requested, how many are being
+released.</p>
+<pre class="codehilite"><code>"rolestatus": {
+ "worker": {
+ "role.instances": "2",
+ "role.requested.instances": "0",
+ "role.failed.starting.instances": "0",
+ "role.actual.instances": "2",
+ "role.releasing.instances": "0",
+ "role.failed.instances": "1"
+ },
+ "slider": {
+ "role.instances": "1",
+ "role.requested.instances": "0",
+ "role.name": "slider",
+ "role.actual.instances": "1",
+ "role.releasing.instances": "0",
+ "role.failed.instances": "0"
+ },
+ "master": {
+ "role.instances": "1",
+ "role.requested.instances": "1",
+ "role.name": "master",
+ "role.failed.starting.instances": "0",
+ "role.actual.instances": "0",
+ "role.releasing.instances": "0",
+ "role.failed.instances": "0"
+ }
+}</code></pre>
+
+
+<h3 id="statusprovider-provider-specific-information"><code>/status/provider</code>: provider-specific information</h3>
+<p>Providers MAY publish information to the <code>/status/provider</code> section.</p>
+<ol>
+<li>There's no restriction on what JSON is permitted in this section.</li>
+<li>Providers may make their own updates to the application state to read and
+write this block -operations that are asynchronous to any status queries.</li>
+</ol>
+<h2 id="statistics-aggregate-statistics"><code>/statistics</code>: aggregate statistics</h2>
+<p>Statistics on the cluster and each role in the cluster </p>
+<p>Better to have a specific <code>/statistics/cluster</code> element,
+and to move the roles' statistics under <code>/statistics/roles</code>:</p>
+<pre class="codehilite"><code>"statistics": {
+ "cluster": {
+ "containers.unknown.completed": 0,
+ "containers.start.completed": 3,
+ "containers.live": 1,
+ "containers.start.failed": 0,
+ "containers.failed": 0,
+ "containers.completed": 0,
+ "containers.surplus": 0
+ },
+ "roles": {
+ "worker": {
+ "containers.start.completed": 0,
+ "containers.live": 2,
+ "containers.start.failed": 0,
+ "containers.active.requests": 0,
+ "containers.failed": 0,
+ "containers.completed": 0,
+ "containers.desired": 2,
+ "containers.requested": 0
+ },
+ "master": {
+ "containers.start.completed": 0,
+ "containers.live": 1,
+ "containers.start.failed": 0,
+ "containers.active.requests": 0,
+ "containers.failed": 0,
+ "containers.completed": 0,
+ "containers.desired": 1,
+ "containers.requested": 0
+ }
+ }
+},</code></pre>
+
+
+<p><code>/statistics/cluster</code> provides aggregate statistics for the entire cluster.</p>
+<p>Under <code>/statistics/roles</code> MUST come an entry for each role in the cluster.</p>
+<p>All simple values in statistics section are integers.</p>
+<h3 id="clientproperties"><code>/clientProperties</code></h3>
+<p>The <code>/clientProperties</code> section contains key-val pairs of type
+string, the expectation being this is where providers can insert specific
+single attributes for client applications.</p>
+<p>These values can be converted to application-specific files on the client,
+in code -as done today in the Slider CLI-, or via template expansion (beyond
+the scope of this document.</p>
+<h3 id="clientfiles"><code>/clientfiles</code></h3>
+<p>This section list all files that an application instance MAY generate
+for clients, along with with a description.</p>
+<pre class="codehilite"><code>"/clientfiles/hbase-site.xml": "site information for HBase"
+"/clientfiles/log4.properties": "log4.property file"</code></pre>
+
+
+<p>Client configuration file retrieval is by other means; this
+status operation merely lists files that are available;</p>
+ </div>
+
+ <div id="footer">
+ <a alt="Apache Software Foundation" href="http://www.apache.org">
+ <img id="asf-logo" alt="Apache Software Foundation" src="/images/feather-small.gif" width="100">
+ </a>
+ <div class="copyright">
+ <p>
+ Copyright © 2011-2014 The Apache Software Foundation, Licensed under
+ the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.
+ Apache Slider, Slider, Apache, the Apache feather logo, and the Apache Incubator
+ project logo are trademarks of the <a href="http://www.apache.org">Apache Software Foundation</a>.<br />
+ Site created with <a href="http://getbootstrap.com/">Bootstrap</a> including icons from <a href="http://glyphicons.com/">GLYPHICONS</a> and <a href="http://fontawesome.io/">Font Awesome</a>.
+ </p>
+ </div>
+ </div>
+ </div>
+
+</div>
+</div>
+
+<script type="text/javascript">
+
+</script>
+</body>
+</html>
Propchange: incubator/slider/site/content/docs/configuration/specification.html
------------------------------------------------------------------------------
svn:eol-style = native