You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@slider.apache.org by sm...@apache.org on 2014/07/10 02:13:38 UTC
svn commit: r1609347 - in /incubator/slider/site/trunk:
content/docs/getting_started.md
content/docs/slider_specs/application_instance_configuration.md
content/docs/slider_specs/index.md templates/nav.html
Author: smohanty
Date: Thu Jul 10 00:13:37 2014
New Revision: 1609347
URL: http://svn.apache.org/r1609347
Log:
SLIDER-121 manually generate slider site from CMS content for now (preparation for next release)
Modified:
incubator/slider/site/trunk/content/docs/getting_started.md
incubator/slider/site/trunk/content/docs/slider_specs/application_instance_configuration.md
incubator/slider/site/trunk/content/docs/slider_specs/index.md
incubator/slider/site/trunk/templates/nav.html
Modified: incubator/slider/site/trunk/content/docs/getting_started.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/getting_started.md?rev=1609347&r1=1609346&r2=1609347&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/docs/getting_started.md (original)
+++ incubator/slider/site/trunk/content/docs/getting_started.md Thu Jul 10 00:13:37 2014
@@ -22,7 +22,7 @@
The following provides the steps required for setting up a cluster and deploying a YARN hosted application using Slider.
-* [Prerequisites](#sysreqs)
+* [System Requirements](#sysreqs)
* [Setup the Cluster](#setup)
@@ -34,13 +34,10 @@ The following provides the steps require
* [Deploy Slider Resources](#deploy)
-* [Download Sample Application Packages](#downsample)
+* [Create Application Packages](#downsample)
* [Install, Configure, Start and Verify Sample Application](#installapp)
-* [Appendix A: Storm Sample Application Specifications](#appendixa)
-
-* [Appendix B: HBase Sample Application Specifications](#appendixb)
## <a name="sysreqs"></a>System Requirements
@@ -52,6 +49,8 @@ The Slider deployment has the following
* Oracle JDK 1.7 (64-bit)
+* Python 2.7
+
## <a name="setup"></a>Setup the Cluster
After setting up your Hadoop cluster (using Ambari or other means) with the
@@ -65,7 +64,7 @@ containers on a single host. In `yarn-si
</tr>
<tr>
<td>yarn.scheduler.minimum-allocation-mb</td>
- <td>>= 256</td>
+ <td>>= 256 (ensure that YARN can allocate sufficient number of containers)</td>
</tr>
<tr>
<td>yarn.nodemanager.delete.debug-delay-sec</td>
@@ -82,14 +81,16 @@ There are other options detailed in the
Slider releases are available at
[https://www.apache.org/dyn/closer.cgi/incubator/slider](https://www.apache.org/dyn/closer.cgi/incubator/slider).
+*Or, you can build it as described below.*
+
## <a name="build"></a>Build Slider
-* From the top level directory, execute `mvn clean install -DskipTests`
+* From the top level directory, execute `mvn clean site:site site:stage package -DskipTests`
* Use the generated compressed tar file in slider-assembly/target directory (e.g. slider-0.30.0-all.tar.gz) for the subsequent steps
## <a name="install"></a>Install Slider
-Follow the following steps to expand/install Slider:
+Slider is installed on a client machine that can access the hadoop cluster. Follow the following steps to expand/install Slider:
mkdir ${slider-install-dir*;
@@ -141,11 +142,11 @@ Execute:
${slider-install-dir}/slider-0.30.0/bin/slider version
-Ensure there are no errors and you can see "Compiled against Hadoop 2.4.0"
+Ensure there are no errors and you can see "Compiled against Hadoop 2.4.0". **This ensures that Slider is installed correctly.**
## <a name="deploy"></a>Deploy Slider Resources
-Ensure that all file folders are accessible to the user creating the application instance. The example assumes "yarn" to be that user.
+Ensure that all file folders are accessible to the user creating the application instance. The example assumes "yarn" to be that user. The HDFS folder "/slider" is a suggestion and the default application packages and associated configurations assume this path while accessing the slider-agent tarball.
### Create HDFS root folder for Slider
@@ -172,11 +173,15 @@ Perform the following steps to create th
## <a name="downsample"></a>Create Application Packages
-Slider "master" branch has three sample application packages available use with Slider:
+Few pre-created application packages are available at: TBD
+
+Slider "master" branch has three sample application packages available for use with Slider:
* app-packages/hbase: The README.txt file describes how to create a Slider Application Package for HBase. By default it will create a package for HBase 0.98.3 but you can create the same for other versions.
+
* app-packages/accumulo: The README.txt file describes how to create a Slider Application Package for Accumulo.
-* app-packages/storm: TBD
+
+* app-packages/storm: The README.txt describes how to create the Slider Application package for Storm.
Create one or more Slider application packages and follow the steps below to install them.
@@ -215,12 +220,16 @@ All this information goes into the **Res
Store the Resource Spec file on your local disk (e.g. `/tmp/resources.json`).
+[Appendix B](#appendixb) provides some sample resources.json files.
+
#### <a name="appconfig"></a>Application Configuration
Alongside the Resource Spec there is the **Application Configuration** file ("App Config") which includes parameters that are specific to the application, rather than YARN. The App Config is a file that contains all application configuration. This configuration is applied to the default configuration provided by the application definition and then handed off to the associated component agent.
For example, the heap sizes of the JVMs, The App Config defines the configuration details **specific to the application and component** instances. For HBase, this includes any values for the *to-be-generated *hbase-site.xml file, as well as options for individual components, such as their heap size.
+[Appendix C](#appendixc) provides a sample appConfig.json file.
+
### <a name="start"></a>Start the Application
Once the steps above are completed, the application can be started through the **Slider Command Line Interface (CLI)**.
@@ -299,9 +308,9 @@ Once started, applications can be frozen
./slider flex cl1 --component worker 5
-# <a name="appendixa"></a>Appendix A: Debugging Slider-Agent
+# <a name="appendixa"></a>Appendix A: Debugging Slider-Agent
-### Create and deploy Slider Agent configuration
+## Create and deploy Slider Agent configuration
Create an agent config file (agent.ini) based on the sample available at:
@@ -362,5 +371,105 @@ Once created, deploy the agent.ini file
Modify the --template json file (appConfig.json) and add the location of the agent.ini file.
"agent.conf": "/slider/agent/conf/agent.ini"
+
+# <a name="appendixb"></a>Appendix B: Resource template (resources.json)
+
+This sample is suitable for HBase application.
+
+ {
+ "schema": "http://example.org/specification/v2.0.0",
+ "metadata": {
+ },
+ "global": {
+ },
+ "components": {
+ "HBASE_MASTER": {
+ "yarn.role.priority": "1",
+ "yarn.component.instances": "1",
+ "yarn.memory": "256"
+ },
+ "slider-appmaster": {
+ },
+ "HBASE_REGIONSERVER": {
+ "yarn.role.priority": "2",
+ "yarn.component.instances": "1",
+ "yarn.memory": "256"
+ }
+ }
+ }
+
+# <a name="appendixc"></a>Appendix C: Config template (appCofig.json)
+
+This sample is suitable for HBase application. See [Application Instance Configuration](/doc/slider_specs/application_instance_configuration.html) for detailed explanation of how to create an appConfig.json file.
+
+ {
+ "schema": "http://example.org/specification/v2.0.0",
+ "metadata": {
+ },
+ "global": {
+ "application.def": "${app.package.name}.zip",
+ "create.default.zookeeper.node": "true",
+ "config_types": "core-site,hdfs-site,hbase-site",
+ "java_home": "/usr/jdk64/jdk1.7.0_45",
+ "package_list": "files/hbase-${hbase.version}-bin.tar.gz",
+ "site.global.app_user": "yarn",
+ "site.global.app_log_dir": "${AGENT_LOG_ROOT}/app/log",
+ "site.global.app_pid_dir": "${AGENT_WORK_ROOT}/app/run",
+ "site.global.app_root": "${AGENT_WORK_ROOT}/app/install/hbase-${hbase.version}",
+ "site.global.app_install_dir": "${AGENT_WORK_ROOT}/app/install",
+ "site.global.hbase_master_heapsize": "1024m",
+ "site.global.hbase_regionserver_heapsize": "1024m",
+ "site.global.hbase_instance_name": "instancename",
+ "site.global.hbase_root_password": "secret",
+ "site.global.user_group": "hadoop",
+ "site.global.security_enabled": "false",
+ "site.global.monitor_protocol": "http",
+ "site.global.ganglia_server_host": "${NN_HOST}",
+ "site.global.ganglia_server_port": "8667",
+ "site.global.ganglia_server_id": "Application1",
+ "site.hbase-site.hbase.hstore.flush.retries.number": "120",
+ "site.hbase-site.hbase.client.keyvalue.maxsize": "10485760",
+ "site.hbase-site.hbase.hstore.compactionThreshold": "3",
+ "site.hbase-site.hbase.rootdir": "${DEFAULT_DATA_DIR}/data",
+ "site.hbase-site.hbase.stagingdir": "${DEFAULT_DATA_DIR}/staging",
+ "site.hbase-site.hbase.regionserver.handler.count": "60",
+ "site.hbase-site.hbase.regionserver.global.memstore.lowerLimit": "0.38",
+ "site.hbase-site.hbase.hregion.memstore.block.multiplier": "2",
+ "site.hbase-site.hbase.hregion.memstore.flush.size": "134217728",
+ "site.hbase-site.hbase.superuser": "yarn",
+ "site.hbase-site.hbase.zookeeper.property.clientPort": "2181",
+ "site.hbase-site.hbase.regionserver.global.memstore.upperLimit": "0.4",
+ "site.hbase-site.zookeeper.session.timeout": "30000",
+ "site.hbase-site.hbase.tmp.dir": "${AGENT_WORK_ROOT}/work/app/tmp",
+ "site.hbase-site.hbase.local.dir": "${hbase.tmp.dir}/local",
+ "site.hbase-site.hbase.hregion.max.filesize": "10737418240",
+ "site.hbase-site.hfile.block.cache.size": "0.40",
+ "site.hbase-site.hbase.security.authentication": "simple",
+ "site.hbase-site.hbase.defaults.for.version.skip": "true",
+ "site.hbase-site.hbase.zookeeper.quorum": "${ZK_HOST}",
+ "site.hbase-site.zookeeper.znode.parent": "${DEF_ZK_PATH}",
+ "site.hbase-site.hbase.hstore.blockingStoreFiles": "10",
+ "site.hbase-site.hbase.hregion.majorcompaction": "86400000",
+ "site.hbase-site.hbase.security.authorization": "false",
+ "site.hbase-site.hbase.cluster.distributed": "true",
+ "site.hbase-site.hbase.hregion.memstore.mslab.enabled": "true",
+ "site.hbase-site.hbase.client.scanner.caching": "100",
+ "site.hbase-site.hbase.zookeeper.useMulti": "true",
+ "site.hbase-site.hbase.regionserver.info.port": "0",
+ "site.hbase-site.hbase.master.info.port": "${HBASE_MASTER.ALLOCATED_PORT}",
+ "site.hbase-site.hbase.regionserver.port": "0"
+ },
+ "components": {
+ "HBASE_MASTER": {
+ },
+ "slider-appmaster": {
+ "jvm.heapsize": "256M"
+ },
+ "HBASE_REGIONSERVER": {
+ }
+ }
+ }
+
+
Modified: incubator/slider/site/trunk/content/docs/slider_specs/application_instance_configuration.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/slider_specs/application_instance_configuration.md?rev=1609347&r1=1609346&r2=1609347&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/docs/slider_specs/application_instance_configuration.md (original)
+++ incubator/slider/site/trunk/content/docs/slider_specs/application_instance_configuration.md Thu Jul 10 00:13:37 2014
@@ -55,6 +55,7 @@ An appConfig.json contains the applicati
"site.hbase-site.hbase.tmp.dir": "${AGENT_WORK_ROOT}/work/app/tmp",
"site.hbase-site.hbase.master.info.port": "${HBASE_MASTER.ALLOCATED_PORT}",
"site.hbase-site.hbase.regionserver.port": "0",
+ "site.hbase-site.zookeeper.znode.parent": "${DEF_ZK_PATH}",
"site.core-site.fs.defaultFS": "${NN_URI}",
"site.hdfs-site.dfs.namenode.https-address": "${NN_HOST}:50470",
@@ -64,7 +65,7 @@ An appConfig.json contains the applicati
appConf.json allows you to pass in arbitrary set of configuration that Slider will forward to the application component instances.
-**Variable naming convention**
+## Variable naming convention
In order to understand how the naming convention work, lets look at how the config is passed on to component commands. Slider agent recevies a structured bag of commands as input for all commands, INSTALL, CONFIGURE, START, etc. The command includes a section "configuration" which has config properties arranged into named property bags.
* Variables of the form `site.xx.yy` translates to variables by the name `yy` within the group `xx` and are typically converted to site config files by the name `xx` containing variable `yy`. For example, `"site.hbase-site.hbase.regionserver.port":""` will be sent to the Slider-Agent as `"hbase-site" : { "hbase.regionserver.port": ""}` and app definition scripts can access all variables under `hbase-site` as a single property bag.
@@ -74,12 +75,16 @@ In order to understand how the naming co
**slider variables**
* Any config not of the form `site.xx.yy` are consumed by Slider itself. Some of the manadatory configuration are:
- * `agent.conf`: location of the agent config file (typically, "/slider/agent/conf/agent.ini")
- * `application.def`: location of the application definition package (typically, "/slider/hbase_v096.zip")
+ * `application.def`: location of the application definition package (e.g., "/slider/hbase_v096.zip")
* `config_types`: list of config types sent to the containers (e.g. "core-site,hdfs-site,hbase-site")
* `java_home`: java home path (e.g. "/usr/jdk64/jdk1.7.0_45")
* `package_list`: location of the package relative to the root where AppPackage is installed (e.g. "files/hbase-0.96.1-hadoop2-bin.tar.gz"
+ OPTIONAL
+ * `create.default.zookeeper.node`: whether the application needs default ZK node
+ * `agent.conf`: location of the agent config file (typically, "/slider/agent/conf/agent.ini")
+
+
**dynamically allocated ports**
Apps can ask port to be dynamically assigned by Slider or they can leave it as "0". If there is a need for advertising any listening endpoint then the ports can be marked such.
@@ -100,5 +105,38 @@ Ganglia server lifecycle is not controll
All three variable values are user provided. It is also expected that a gmond server is available on host gangliaserver.my.org listening for metrics at port 8663 and is named "HBaseApplicationCluster3". Its the reponsibility of the ganglia server admin to ensure that the server is unique and is only receving metrics from the application instance.
+## Available environment variable
+
+<table>
+ <tr>
+ <td>Property</td>
+ <td>Description</td>
+ </tr>
+ <tr>
+ <td>AGENT_LOG_ROOT</td>
+ <td>The root folder for the log</td>
+ </tr>
+ <tr>
+ <td>AGENT_WORK_ROOT</td>
+ <td>The root working folder</td>
+ </tr>
+ <tr>
+ <td>NN_URI</td>
+ <td>Name node URI given to Slider</td>
+ </tr>
+ <tr>
+ <td>NN_HOST</td>
+ <td>Name node host</td>
+ </tr>
+ <tr>
+ <td><COMPONENT_NAME>.ALLOCATED_PORT</td>
+ <td>Ask Slider to allocate a port</td>
+ </tr>
+ <tr>
+ <td>DEF_ZK_PATH</td>
+ <td>Default ZK path created by Slider if create.default.zookeeper.node is true</td>
+ </tr>
+</table>
+
Modified: incubator/slider/site/trunk/content/docs/slider_specs/index.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/slider_specs/index.md?rev=1609347&r1=1609346&r2=1609347&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/docs/slider_specs/index.md (original)
+++ incubator/slider/site/trunk/content/docs/slider_specs/index.md Thu Jul 10 00:13:37 2014
@@ -45,9 +45,9 @@ The entry points to leverage Slider are:
- [Slider AppPackage](creating_app_definitions.html) Overview of how to create an Slider AppPackage.
- [Specifications for AppPackage](application_package.html) Describes the structure of an AppPackage
- [Specifications for Application Definition](application_definition.html) How to write metainfo.xml?
-- [Specifications for Configuration](application_configuration.html) How to create a template config file for an app?
- [Specification of Resources](resource_specification.html) How to write a resource spec for an app?
- [Specifications InstanceConfiguration](application_instance_configuration.html) How to write a template config for an app?
- [Guidelines for Clients and Client Applications](canonical_scenarios.html)
+- [Specifications for Configuration](application_configuration.html) Default application configuration?
- [Documentation for "General Developer Guidelines"](/developing/index.html)
Modified: incubator/slider/site/trunk/templates/nav.html
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/templates/nav.html?rev=1609347&r1=1609346&r2=1609347&view=diff
==============================================================================
--- incubator/slider/site/trunk/templates/nav.html (original)
+++ incubator/slider/site/trunk/templates/nav.html Thu Jul 10 00:13:37 2014
@@ -70,7 +70,7 @@
<li id="nav_use_client_config"><a href="/docs/client-configuration.html">Client Configuration</a></li>
<li id="nav_use_exitcodes"><a href="/docs/exitcodes.html">Client Exit Codes</a></li>
<li id="nav_use_security"><a href="/docs/security.html">Security</a></li>
-<li id="nav_use_app_package"><a href="/docs/slider_specs/index.html">App Packaging</a></li>
+<li id="nav_use_app_package"><a href="/docs/slider_specs/index.html">Creating App Packages</a></li>
<li id="nav_use_app_configuration"><a href="/docs/configuration/index.html">App Configuration Model</a></li>
<li id="nav_use_debugging"><a href="/docs/debugging.html">Debugging</a></li>
<li id="nav_use_troubleshooting"><a href="/docs/troubleshooting.html">Troubleshooting</a></li>