You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@slider.apache.org by sm...@apache.org on 2014/05/28 04:21:15 UTC
[10/44] git commit: Documentation changes to add more details of app
creation.
Documentation changes to add more details of app creation.
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/f3dd43f3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/f3dd43f3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/f3dd43f3
Branch: refs/heads/master
Commit: f3dd43f31a7787136580f58506219566641e5146
Parents: ad87924
Author: Sumit Mohanty <sm...@hortonworks.com>
Authored: Wed May 21 17:36:18 2014 -0700
Committer: Sumit Mohanty <sm...@hortonworks.com>
Committed: Wed May 21 17:36:18 2014 -0700
----------------------------------------------------------------------
.../slider_specs/application_definition.md | 12 +--
.../application_instance_configuration.md | 92 +++++++++++++++-----
.../slider_specs/creating_app_definitions.md | 89 +++----------------
src/site/markdown/slider_specs/index.md | 14 +--
.../slider_specs/resource_specification.md | 46 +++++-----
5 files changed, 113 insertions(+), 140 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/f3dd43f3/src/site/markdown/slider_specs/application_definition.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/slider_specs/application_definition.md b/src/site/markdown/slider_specs/application_definition.md
index 52f8106..6e417d2 100644
--- a/src/site/markdown/slider_specs/application_definition.md
+++ b/src/site/markdown/slider_specs/application_definition.md
@@ -17,7 +17,7 @@
# Application Definition
-App definition is a declarative definition of a YARN application describing its content. The AppDefinition is used in conjunction with the [AppPackage](application_package.md).
+App definition is a declarative definition of a YARN application describing its content. The AppDefinition is used in conjunction with the [AppPackage](application_package.md). Application definition is an XML file and is packaged as metainfo.xml.
## Structure
@@ -180,13 +180,3 @@ Each exportGroup contains one or more exports.
</application>
</metainfo>
-
-
-## Open Questions
-
-1. Applications may need some information from other applications or base services such as ZK, YARN, HDFS. Additionally, they may need a dedicated ZK node, a HDFS working folder, etc. How do we capture this requirement? There needs to be a well-known way to ask for these information e.g. fs.default.name, zk_hosts.
-
-2. Similar to the above there are common parameters such as `JAVA_HOME` and other environment variables. Application should be able to refer to these parameters and Slider should be able to provide them.
-
-3. Composite application definition: Composite application definition would require a spec that refers to this spec and binds multiple applications together.
-
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/f3dd43f3/src/site/markdown/slider_specs/application_instance_configuration.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/slider_specs/application_instance_configuration.md b/src/site/markdown/slider_specs/application_instance_configuration.md
index dad7f4e..764d3ce 100644
--- a/src/site/markdown/slider_specs/application_instance_configuration.md
+++ b/src/site/markdown/slider_specs/application_instance_configuration.md
@@ -23,38 +23,82 @@ Instance configuration is a JSON formatted doc in the following form:
{
- "configurations": {
- "app-global-config": {
- },
- "config-type-1": {
- },
- "config-type-2": {
- },
- }
+ "schema": "http://example.org/specification/v2.0.0",
+ "metadata": {
+ },
+ "global": {
+ },
}
+An appConfig.json contains the application configuration. The sample below shows configuration for HBase.
-The configuration overrides are organized in a two level structure where name-value pairs are grouped on the basis of config types they belong to. App instantiator can provide arbitrary custom name-value pairs within a config type defined in the AppPackage or can create a completely new config type that does not exist in the AppAPackage. The interpretation of the configuration is entirely up to the command implementations present in the AppPackage. Slider will simply merge the configs with the InstanceConfiguration being higher priority than that default configuration and hand it off to the app commands.
+ {
+ "schema" : "http://example.org/specification/v2.0.0",
+ "metadata" : {
+ },
+ "global" : {
+ "config_types": "core-site,hdfs-site,hbase-site",
+
+ "java_home": "/usr/jdk64/jdk1.7.0_45",
+ "package_list": "files/hbase-0.96.1-hadoop2-bin.tar",
+
+ "site.global.app_user": "yarn",
+ "site.global.app_log_dir": "${AGENT_LOG_ROOT}/app/log",
+ "site.global.app_pid_dir": "${AGENT_WORK_ROOT}/app/run",
+ "site.global.security_enabled": "false",
+
+ "site.hbase-site.hbase.hstore.flush.retries.number": "120",
+ "site.hbase-site.hbase.client.keyvalue.maxsize": "10485760",
+ "site.hbase-site.hbase.hstore.compactionThreshold": "3",
+ "site.hbase-site.hbase.rootdir": "${NN_URI}/apps/hbase/data",
+ "site.hbase-site.hbase.tmp.dir": "${AGENT_WORK_ROOT}/work/app/tmp",
+ "site.hbase-site.hbase.master.info.port": "${HBASE_MASTER.ALLOCATED_PORT}",
+ "site.hbase-site.hbase.regionserver.port": "0",
+
+ "site.core-site.fs.defaultFS": "${NN_URI}",
+ "site.hdfs-site.dfs.namenode.https-address": "${NN_HOST}:50470",
+ "site.hdfs-site.dfs.namenode.http-address": "${NN_HOST}:50070"
+ }
+ }
-A sample config for hbase may be as follows:
+appConf.json allows you to pass in arbitrary set of configuration that Slider will forward to the application component instances.
+**Variable naming convention**
+In order to understand how the naming convention work, lets look at how the config is passed on to component commands. Slider agent recevies a structured bag of commands as input for all commands, INSTALL, CONFIGURE, START, etc. The command includes a section "configuration" which has config properties arranged into named property bags.
+* Variables of the form `site.xx.yy` translates to variables by the name `yy` within the group `xx` and are typically converted to site config files by the name `xx` containing variable `yy`. For example, `"site.hbase-site.hbase.regionserver.port":""` will be sent to the Slider-Agent as `"hbase-site" : { "hbase.regionserver.port": ""}` and app definition scripts can access all variables under `hbase-site` as a single property bag.
+* Similarly, `site.core-site.fs.defaultFS` allows you to pass in the default fs. *This specific variable is automatically made available by Slider but its shown here as an example.*
+* Variables of the form `site.global.zz` are sent in the same manner as other site variables except these variables are not expected to get translated to a site xml file. Usually, variables needed for template or other filter conditions (such as security_enabled = true/false) can be sent in as "global variable".
- {
- "configurations": {
- "hbase-log4j": {
- "log4j.logger.org.apache.zookeeper": "INFO",
- "log4j.logger.org.apache.hadoop.hbase": "DEBUG"
- },
- "hbase-site": {
- "hbase.hstore.flush.retries.number": "120",
- "hbase.regionserver.info.port": "",
- "hbase.master.info.port": "60010"
- }
- }
-
+**slider variables**
+
+* Any config not of the form `site.xx.yy` are consumed by Slider itself. Some of the manadatory configuration are:
+ * `agent.conf`: location of the agent config file (typically, "/slider/agent/conf/agent.ini")
+ * `application.def`: location of the application definition package (typically, "/slider/hbase_v096.zip")
+ * `config_types`: list of config types sent to the containers (e.g. "core-site,hdfs-site,hbase-site")
+ * `java_home`: java home path (e.g. "/usr/jdk64/jdk1.7.0_45")
+ * `package_list`: location of the package relative to the root where AppPackage is installed (e.g. "files/hbase-0.96.1-hadoop2-bin.tar.gz"
+
+**dynamically allocated ports**
+
+Apps can ask port to be dynamically assigned by Slider or they can leave it as "0". If there is a need for advertising any listening endpoint then the ports can be marked such.
+
+For example, HBase master info port needs to be advertised so that jmx endpoint can be accessed. This is indicated by using a special value, of the form, ${COMPONENT_NAME.ALLOCATED_PORT}. E.g. "site.hbase-site.hbase.master.info.port": "${HBASE_MASTER.ALLOCATED_PORT}"
+
+[Application Definition](application_definition.md) describes how to advertise arbitrary set of properties that are dynamically finalized when application is activated.
+
+**configuraing an app for ganglia metrics**
+
+There is no set guideline for doing so. How an application emits metrics and how the metrics are emitted to the right place is completely defined by the application. In the following example, we hso how HBase app is configured to emit metrics to a ganglia server.
+
+Ganglia server lifecycle is not controlled by the app instance. So the app instance only needs to know where to emit the metrics. This is achieved by three global variables
+
+* "site.global.ganglia_server_host": "gangliaserver.my.org"
+* "site.global.ganglia_server_port": "8663"
+* "site.global.ganglia_server_id": "HBaseApplicationCluster3"
+
+All three variable values are user provided. It is also expected that a gmond server is available on host gangliaserver.my.org listening for metrics at port 8663 and is named "HBaseApplicationCluster3". Its the reponsibility of the ganglia server admin to ensure that the server is unique and is only receving metrics from the application instance.
-The above config overwrites few parameters in hbase-site and hbase-log4j files. Several config properties such as `hbase.zookeeper.quorum` for hbase may not be known to the user at the time of app instantiation. These configurations will be provided by the Slider infrastructure in a well-known form so that the app implementation can read and set them while instantiating component instances..
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/f3dd43f3/src/site/markdown/slider_specs/creating_app_definitions.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/slider_specs/creating_app_definitions.md b/src/site/markdown/slider_specs/creating_app_definitions.md
index 6015eb1..b89df62 100644
--- a/src/site/markdown/slider_specs/creating_app_definitions.md
+++ b/src/site/markdown/slider_specs/creating_app_definitions.md
@@ -15,7 +15,7 @@
limitations under the License.
-->
-# Define and use Slider AppPackage
+# Slider AppPackage
Slider AppPackages are a declarative definition of an application for application management. AppPackage is not a packaging scheme for application binaries and artifacts. Tarball, zip files, rpms etc. are available for that purpose. Instead AppPackage includes the application binaries along with other artifacts necessary for application management.
@@ -31,18 +31,16 @@ For example:
yarn 9085 -- bash /hadoop/yarn/local/usercache/yarn/appcache/application_1397675825552_0011/ ... internal_start regionserver
yarn 9114 -- /usr/jdk64/jdk1.7.0_45/bin/java -Dproc_regionserver -XX:OnOutOfMemoryError=...
-Shows three processes, the Slider-Agent process, the bash script to start HBase Region Server and the HBase Region server itself. Three of these together constitute the container.
+The above list shows three processes, the Slider-Agent process, the bash script to start HBase Region Server and the HBase Region server itself. *Three of these together constitute the container*.
## Using an AppPackage
The following command creates an HBase application using the AppPackage for HBase.
- ./slider create cl1 --zkhosts zk1,zk2 --image hdfs://NN:8020/slider/agent/slider-agent-0.21.tar --option agent.conf hdfs://NN:8020/slider/agent/conf/agent.ini --template /work/appConf.json --resources /work/resources.json --option application.def hdfs://NN:8020/slider/hbase_v096.zip
+ ./slider create cl1 --image hdfs://NN:8020/slider/agent/slider-agent.tar.gz --template /work/appConf.json --resources /work/resources.json
Lets analyze various parameters from the perspective of app creation:
* `--image`: its the slider agent tarball
-* `--option agent.conf`: the configuration file for the agent instance
-* `--option app.def`: app def (AppPackage)
* `--template`: app configuration
* `--resources`: yarn resource requests
* … other parameters are described in accompanying docs.
@@ -50,13 +48,13 @@ Lets analyze various parameters from the perspective of app creation:
### AppPackage
The structure of an AppPackage is described at [AppPackage](application_package.md).
-In the enlistment there are three example AppPackages
+In the enlistment, there are three example AppPackages:
* `app-packages/hbase-v0_96`
* `app-packages/accumulo-v1_5`
* `app-packages/storm-v0_91`
-The application zip file, containing the binaries/artifacts of the application itself is a component within the AppPackage. They are:
+The above folders, with minor edits, can be packaged as *zip* files to get the corresponding AppPackages. The application tarball file, containing the binaries/artifacts of the application itself is a component within the AppPackage. They are:
* For hbase - `app-packages/hbase-v0_96/package/files/hbase-0.96.1-hadoop2-bin.tar.gz.REPLACE`
* For accumulo - `app-packages/accumulo-v1_5/package/files/accumulo-1.5.1-bin.tar.gz.REPLACE`
@@ -64,8 +62,6 @@ The application zip file, containing the binaries/artifacts of the application i
**They are placehoder files**, mostly because the files themselves are too large as well as users are free to use their own version of the package. To create a Slider AppPackage - replace the file with an actual application tarball and then ensure that the metainfo.xml has the correct file name. After that create a zip file using standard zip commands and ensure that the package has the metainfo.xml file at the root folder.
-Sample **resources.json** and **appConfig.json** files are also included in the enlistment.
-
For example:
* cd slider/app-packages/hbase-v0_96
@@ -109,79 +105,18 @@ Archive: hbase_v096.zip
83219742 29 files
```
+Sample **resources.json** and **appConfig.json** files are also included in the enlistment. These are samples and are typically tested on one node test installations.
-### appConf.json
-An appConf.json contains the application configuration. The sample below shows configuration for HBase.
-
-
- {
- "schema" : "http://example.org/specification/v2.0.0",
- "metadata" : {
- },
- "global" : {
- "config_types": "core-site,hdfs-site,hbase-site",
-
- "java_home": "/usr/jdk64/jdk1.7.0_45",
- "package_list": "files/hbase-0.96.1-hadoop2-bin.tar",
-
- "site.global.app_user": "yarn",
- "site.global.app_log_dir": "${AGENT_LOG_ROOT}/app/log",
- "site.global.app_pid_dir": "${AGENT_WORK_ROOT}/app/run",
- "site.global.security_enabled": "false",
-
- "site.hbase-site.hbase.hstore.flush.retries.number": "120",
- "site.hbase-site.hbase.client.keyvalue.maxsize": "10485760",
- "site.hbase-site.hbase.hstore.compactionThreshold": "3",
- "site.hbase-site.hbase.rootdir": "${NN_URI}/apps/hbase/data",
- "site.hbase-site.hbase.tmp.dir": "${AGENT_WORK_ROOT}/work/app/tmp",
- "site.hbase-site.hbase.regionserver.port": "0",
-
- "site.core-site.fs.defaultFS": "${NN_URI}",
- "site.hdfs-site.dfs.namenode.https-address": "${NN_HOST}:50470",
- "site.hdfs-site.dfs.namenode.http-address": "${NN_HOST}:50070"
- }
- }
-appConf.jso allows you to pass in arbitrary set of configuration that Slider will forward to the application component instances.
+### --template appConfig.json
+An appConfig.json contains the application configuration. See [Specifications InstanceConfiguration](application_instance_configuration.md) for details on how to create a template config file. The enlistment includes sample config files for HBase, Accumulo, and Storm.
-* Variables of the form `site.xx.yy` translates to variables by the name `yy` within the group `xx` and are typically converted to site config files by the name `xx` containing variable `yy`. For example, `"site.hbase-site.hbase.regionserver.port":""` will be sent to the Slider-Agent as `"hbase-site" : { "hbase.regionserver.port": ""}` and app def scripts can access all variables under `hbase-site` as a single property bag.
-* Similarly, `site.core-site.fs.defaultFS` allows you to pass in the default fs. *This specific variable is automatically made available by Slider but its shown here as an example.*
-* Variables of the form `site.global.zz` are sent in the same manner as other site variables except these variables are not expected to get translated to a site xml file. Usually, variables needed for template or other filter conditions (such as security_enabled = true/false) can be sent in as "global variable".
### --resources resources.json
-The resources.json file encodes the Yarn resource count requirement for the application instance.
-
-The components section lists the two application component for an HBase application.
-
-* `wait.heartbeat`: a crude mechanism to control the order of component activation. A heartbeat is ~10 seconds.
-* `role.priority`: each component must be assigned unique priority
-* `component.instances`: number of instances for this component type
-* `role.script`: the script path for the role *a temporary work-around as this will eventually be gleaned from metadata.xml*
-
-Sample:
-
- {
- "schema" : "http://example.org/specification/v2.0.0",
- "metadata" : {
- },
- "global" : {
- },
- "components" : {
- "HBASE_MASTER" : {
- "yarn.role.priority" : "1",
- "yarn.component.instances" : "1"
- },
- "slider-appmaster" : {
- "jvm.heapsize" : "256M"
- },
- "HBASE_REGIONSERVER" : {
- "yarn.role.priority" : "2",
- "yarn.component.instances" : "1"
- }
- }
- }
-
-## Creating AppPackage
+Resource specification is an input to Slider to specify the Yarn resource needs for each component type that belong to the application. [Specification of Resources](resource_specification.html) describes how to write a resource config json file. The enlistment includes sample config files for HBase, Accumulo, and Storm.
+
+
+## Scripting for AppPackage
Refer to [App Command Scripts](writing_app_command_scripts) for details on how to write scripts for a AppPackage. These scripts are in the package/script folder within the AppPackage. *Use the checked in samples for HBase/Storm/Accumulo as reference for script development.*
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/f3dd43f3/src/site/markdown/slider_specs/index.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/slider_specs/index.md b/src/site/markdown/slider_specs/index.md
index d960588..114950c 100644
--- a/src/site/markdown/slider_specs/index.md
+++ b/src/site/markdown/slider_specs/index.md
@@ -41,11 +41,13 @@ Specifications
The entry points to leverage Slider are:
-- [Specifications for AppPackage](application_package.html)
-- [Application Needs](application_needs.html)
-- [Specifications for Application Definition](application_definition.html)
-- [Specifications for Configuration](application_configuration.html)
-- [Specification of Resources](resource_specification.html)
-- [Specifications InstanceConfiguration](application_instance_configuration.html)
+- [Application Needs](application_needs.html) What it takes to be deployable by Slider.
+- [Slider AppPackage](creating_app_definitions.html) Overview of how to create an Slider AppPackage.
+- [Specifications for AppPackage](application_package.html) Describes the structure of an AppPackage
+- [Specifications for Application Definition](application_definition.html) How to write metainfo.xml?
+- [Specifications for Configuration](application_configuration.html) How to create a template config file for an app?
+- [Specification of Resources](resource_specification.html) How to write a resource spec for an app?
+- [Specifications InstanceConfiguration](application_instance_configuration.html) How to write a template config for an app?
- [Guidelines for Clients and Client Applications](canonical_scenarios.html)
- [Documentation for "General Developer Guidelines"](app_developer_guideline.html)
+
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/f3dd43f3/src/site/markdown/slider_specs/resource_specification.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/slider_specs/resource_specification.md b/src/site/markdown/slider_specs/resource_specification.md
index aef2877..6d41f9e 100644
--- a/src/site/markdown/slider_specs/resource_specification.md
+++ b/src/site/markdown/slider_specs/resource_specification.md
@@ -18,34 +18,36 @@
#Resource Specification
Resource specification is an input to Slider to specify the Yarn resource needs for each component type that belong to the application.
+An example resource requirement for an application that has two components "master" and "worker" is as follows. Slider will automatically add the requirements for the AppMaster for the application. This compoent is named "slider-appmaster".
+
Some parameters that can be specified for a component instance include:
-* yarn.memory: amount of memory requried for the component instance
-* env.MALLOC_ARENA_MAX: maximum number of memory pools used, arbitrary environment settings can be provided through format env.NAME_OF_THE_VARIABLE
-* component.instances: number of instances requested
-* component.name: name of the component
-* yarn.vcores: number of vcores requested
+* `yarn.memory`: amount of memory requried for the component instance
+* `yarn.vcores`: number of vcores requested
+* `yarn.role.priority`: each component must be assigned unique priority. Component with higher priority come up earlier than components with lower priority
+* `yarn.component.instances`: number of instances for this component type
-An example resource requirement for an application that has two components "master" and "worker" is as follows. Slider will automatically add the requirements for the AppMaster for the application. This compoent is named "slider".
+Sample:
- "components" : {
- "worker" : {
- "yarn.memory" : "768",
- "yarn.component.instances" : "1",
- "yarn.component.name" : "worker",
- "yarn.vcores" : "1"
+ {
+ "schema" : "http://example.org/specification/v2.0.0",
+ "metadata" : {
},
- "slider" : {
- "yarn.memory" : "256",
- "yarn.component.instances" : "1",
- "yarn.component.name" : "slider",
- "yarn.vcores" : "1"
+ "global" : {
},
- "master" : {
- "yarn.memory" : "1024",
- "yarn.component.instances" : "1",
- "yarn.component.name" : "master",
- "yarn.vcores" : "1"
+ "components" : {
+ "HBASE_MASTER" : {
+ "yarn.role.priority" : "1",
+ "yarn.component.instances" : "1"
+ "yarn.memory" : "768",
+ "yarn.vcores" : "1"
+ },
+ "slider-appmaster" : {
+ },
+ "HBASE_REGIONSERVER" : {
+ "yarn.role.priority" : "2",
+ "yarn.component.instances" : "1"
+ }
}
}